facebookresearch / CompilerGym

Reinforcement learning environments for compiler and program optimization tasks
https://compilergym.ai/
MIT License
906 stars 127 forks source link

[llvm] Add utility functions to split and merge benchmarks. #772

Closed ChrisCummins closed 1 year ago

ChrisCummins commented 1 year ago

This adds three new utility functions for constructing LLVM benchmarks:

def make_benchmark_from_source(
    source: str,
    copt: Optional[List[str]] = None,
    lang: str = "c++",
    system_includes: bool = True,
    timeout: int = 600,
) -> Benchmark:
    ...

def split_benchmark_by_function(
    benchmark: Benchmark, timeout: float = 300
) -> List[Benchmark]:
    ...

def merge_benchmarks(
    benchmarks: List[Benchmark], timeout: float = 300
) -> Benchmark:
   ...

Credit to @hughleat for llvm-extract-one, an extension of LLVM's llvm-extract that enables functions to be extracted by an integer index rather than by name. This enables extracting anonymous functions.

Todo before land

codecov-commenter commented 1 year ago

Codecov Report

Merging #772 (19da5e6) into development (b017c81) will increase coverage by 0.32%. The diff coverage is 94.91%.

Additional details and impacted files [![Impacted file tree graph](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/graphs/tree.svg?width=650&height=150&src=pr&token=2C4Q4HC12K&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch)](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) ```diff @@ Coverage Diff @@ ## development #772 +/- ## =============================================== + Coverage 88.56% 88.88% +0.32% =============================================== Files 131 131 Lines 8008 8063 +55 =============================================== + Hits 7092 7167 +75 + Misses 916 896 -20 ``` | [Impacted Files](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) | Coverage Δ | | |---|---|---| | [compiler\_gym/envs/llvm/\_\_init\_\_.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbGx2bS9fX2luaXRfXy5weQ==) | `100.00% <ø> (ø)` | | | [compiler\_gym/envs/llvm/llvm\_benchmark.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbGx2bS9sbHZtX2JlbmNobWFyay5weQ==) | `93.43% <94.54%> (+0.37%)` | :arrow_up: | | [compiler\_gym/datasets/benchmark.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2RhdGFzZXRzL2JlbmNobWFyay5weQ==) | `99.04% <100.00%> (-0.01%)` | :arrow_down: | | [compiler\_gym/third\_party/llvm/\_\_init\_\_.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3RoaXJkX3BhcnR5L2xsdm0vX19pbml0X18ucHk=) | `93.54% <100.00%> (+0.21%)` | :arrow_up: | | [...loop\_tool/service/loop\_tool\_compilation\_session.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbG9vcF90b29sL3NlcnZpY2UvbG9vcF90b29sX2NvbXBpbGF0aW9uX3Nlc3Npb24ucHk=) | `89.86% <0.00%> (-0.68%)` | :arrow_down: | | [compiler\_gym/envs/gcc/service/gcc\_service.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvZ2NjL3NlcnZpY2UvZ2NjX3NlcnZpY2UucHk=) | `95.97% <0.00%> (+0.36%)` | :arrow_up: | | [compiler\_gym/envs/llvm/datasets/poj104.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL2VudnMvbGx2bS9kYXRhc2V0cy9wb2oxMDQucHk=) | `76.81% <0.00%> (+1.44%)` | :arrow_up: | | [compiler\_gym/service/connection.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3NlcnZpY2UvY29ubmVjdGlvbi5weQ==) | `79.28% <0.00%> (+1.94%)` | :arrow_up: | | [compiler\_gym/views/observation\_space\_spec.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3ZpZXdzL29ic2VydmF0aW9uX3NwYWNlX3NwZWMucHk=) | `85.71% <0.00%> (+2.85%)` | :arrow_up: | | [...ompiler\_gym/service/client\_service\_compiler\_env.py](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch#diff-Y29tcGlsZXJfZ3ltL3NlcnZpY2UvY2xpZW50X3NlcnZpY2VfY29tcGlsZXJfZW52LnB5) | `90.98% <0.00%> (+2.93%)` | :arrow_up: | | ... and [1 more](https://codecov.io/gh/facebookresearch/CompilerGym/pull/772/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=facebookresearch) | |