Closed gaperez64 closed 3 years ago
Just a note: the following folders contain parameterized benchmarks (either you missed them or intentionally left out):
tlsf/lift
contains three specs, all are parameterizedtlsf/arbiters_zoo
contains 4 more specs, each is parameterizedarbiters_s4
contain several benchmarks, all are parameterizedamba
contains one parameterized benchmarkAh yes, I missed those. Essentially, I had a script that diff'd files and if I found two that were identical except for the value of a parameter I placed that on the "to parameterize" list. While the ones you mention are indeed parametric, there was only a copy of them in the repo so I did not see them. How about we add this as a TODO for next year :D ?
As a side note, the choice of benchmark families is already visible here (everything non-parametric and all the parametric instances from the csv files in each benchdir will be included). This means that if arbiter_zoo/arbiter.tlsf
and full_arbiter
are merged this will work as well next year (but at the moment, even if the benchmarks are very similar, they will both be included).
Those parameterized benchmarks seemed good to me (lift
is hard, arbiters_zoo
are hard), so it is a bit sad they will be missed this year, but OK, it is not crucial.
Most action seems to be shifting to TLSF benchmarks. I spent a bit of time automating the benchmark-selection process using (Python) scripts @swenjacobs, @5nizza. The main idea is as follows:
If you agree with the changes, we merge this into the main branch and keep that structure for TLSF benchmarks from now on. Then, I will take the current selection and launch the jobs for SYNTCOMP 2021.