Closed atesgoral closed 4 months ago
Increase run-per-level count from 10 to 100 since these run pretty fast.
Instead of just reporting pass rate and moving on, start failing levels when they're not marked as specific and don't pass 100% of runs.
Example output (for this PR) showing a case where a 71% pass rate is deemed acceptable because the solution is marked as specific: https://github.com/atesgoral/hrm-solutions/actions/runs/9802974314/job/27068439475#step:4:21
Rename failing solutions to .asm.skip for now, for triaging:
The unit tests have no value unless we start failing CI on PRs with incomplete or miscategorized solutions.
Looks reasonable to me. The game's handling of solutions is kind of weird, so, being able to have actual random tests is nice.
Increase run-per-level count from 10 to 100 since these run pretty fast.
Instead of just reporting pass rate and moving on, start failing levels when they're not marked as specific and don't pass 100% of runs.
Example output (for this PR) showing a case where a 71% pass rate is deemed acceptable because the solution is marked as specific: https://github.com/atesgoral/hrm-solutions/actions/runs/9802974314/job/27068439475#step:4:21
Rename failing solutions to .asm.skip for now, for triaging:
The unit tests have no value unless we start failing CI on PRs with incomplete or miscategorized solutions.