facebookresearch / CompilerGym

Reinforcement learning environments for compiler and program optimization tasks
https://compilergym.ai/
MIT License
885 stars 123 forks source link

[tests] Minimize the fork regression tests. #754

Closed ChrisCummins closed 1 year ago

ChrisCummins commented 1 year ago

This wrote a simple routine minimize_fork_regression_testcase() that takes as input a forked regression test and iteratively attempts to remove each of the actions that do not prevent the test from failing.

Running this on the five known regression tests simplifies them dratically. In all but one case, they are simplified to a single pre-fork and single post-fork action.

I'll dig more into the details of what's going on in later.

Issue #749.

codecov-commenter commented 1 year ago

Codecov Report

Merging #754 (b1605cd) into development (72fb339) will increase coverage by 1.65%. The diff coverage is n/a.

Additional details and impacted files [![Impacted file tree graph](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/graphs/tree.svg?width=650&height=150&src=pr&token=2C4Q4HC12K&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch)](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) ```diff @@ Coverage Diff @@ ## development #754 +/- ## =============================================== + Coverage 80.46% 82.11% +1.65% =============================================== Files 131 131 Lines 7995 8008 +13 =============================================== + Hits 6433 6576 +143 + Misses 1562 1432 -130 ``` | [Impacted Files](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) | Coverage Δ | | |---|---|---| | [compiler\_gym/util/filesystem.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3V0aWwvZmlsZXN5c3RlbS5weQ==) | `93.33% <0.00%> (-1.41%)` | :arrow_down: | | [compiler\_gym/third\_party/llvm/\_\_init\_\_.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3RoaXJkX3BhcnR5L2xsdm0vX19pbml0X18ucHk=) | `88.33% <0.00%> (+0.19%)` | :arrow_up: | | [compiler\_gym/envs/gcc/service/gcc\_service.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvZ2NjL3NlcnZpY2UvZ2NjX3NlcnZpY2UucHk=) | `95.97% <0.00%> (+0.36%)` | :arrow_up: | | [compiler\_gym/envs/llvm/datasets/cbench.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbGx2bS9kYXRhc2V0cy9jYmVuY2gucHk=) | `74.46% <0.00%> (+0.44%)` | :arrow_up: | | [compiler\_gym/util/runfiles\_path.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3V0aWwvcnVuZmlsZXNfcGF0aC5weQ==) | `91.07% <0.00%> (+1.78%)` | :arrow_up: | | [compiler\_gym/envs/llvm/llvm\_benchmark.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbGx2bS9sbHZtX2JlbmNobWFyay5weQ==) | `51.38% <0.00%> (+2.08%)` | :arrow_up: | | [compiler\_gym/service/connection.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3NlcnZpY2UvY29ubmVjdGlvbi5weQ==) | `75.08% <0.00%> (+2.58%)` | :arrow_up: | | [compiler\_gym/service/service\_cache.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3NlcnZpY2Uvc2VydmljZV9jYWNoZS5weQ==) | `92.10% <0.00%> (+5.26%)` | :arrow_up: | | [compiler\_gym/envs/loop\_tool/\_\_init\_\_.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbG9vcF90b29sL19faW5pdF9fLnB5) | `90.00% <0.00%> (+20.00%)` | :arrow_up: | | [...loop\_tool/service/loop\_tool\_compilation\_session.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/754/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbG9vcF90b29sL3NlcnZpY2UvbG9vcF90b29sX2NvbXBpbGF0aW9uX3Nlc3Npb24ucHk=) | `87.83% <0.00%> (+72.29%)` | :arrow_up: |