atesgoral / hrm-solutions

Human Resource Machine solutions and size/speed hacks
http://atesgoral.github.io/hrm-solutions/
431 stars 178 forks source link

Success checking #378

Closed atesgoral closed 4 months ago

atesgoral commented 4 months ago

Increase run-per-level count from 10 to 100 since these run pretty fast.

Instead of just reporting pass rate and moving on, start failing levels when they're not marked as specific and don't pass 100% of runs.

Example output (for this PR) showing a case where a 71% pass rate is deemed acceptable because the solution is marked as specific: https://github.com/atesgoral/hrm-solutions/actions/runs/9802974314/job/27068439475#step:4:21

Rename failing solutions to .asm.skip for now, for triaging:

The unit tests have no value unless we start failing CI on PRs with incomplete or miscategorized solutions.

clarfonthey commented 4 months ago

Looks reasonable to me. The game's handling of solutions is kind of weird, so, being able to have actual random tests is nice.