A collection of useful program synthesis benchmarks. Each folder contains a different benchmark and a README.
A benchmark has:
Optional:
HerbBenchmarks is still not yet complete and is lacking crucial benchmarking functionality. However, if you want to test on a single problem and grammar, you can do the following
Select your favourite benchmark, we use the string transformation benchmark from the SyGuS challenge:
using HerbSpecification, HerbGrammar
using HerbBenchmarks.PBE_SLIA_Track_2019
# The id has to be matching
grammar = PBE_SLIA_Track_2019.grammar_11604909
problem = PBE_SLIA_Track_2019.problem_11604909
# Print out the grammar and problem in readable format
println("grammar:", grammar)
println("problem:", problem.examples)
For some benchmarks there is only a single grammar for all problems.