Large study and results of different SAT-based samplers:
other: Unigen3, CMS, STS (new!)
over different data:
Pre-built Docker image:
docker run -it -v $(pwd):/home/usampling-exp macher/usampling:squashed /bin/bash -c 'cd /home/usampling-exp/; python3 usampling-experiments.py -t 1 --unigen2 -flas Benchmarks'
for Unigen2 sampler, timeout 1 second, and only formulas contained in the Benchmarks folderwe're planning to provide the script to build the Docker image
Requirements:
docker run -it -v $(pwd):/home/usampling-exp:z macher/usampling:squashed /bin/bash
for developping... you can edit files that are bound to the Docker file. And experiments with procedures/samplers inside the Docker.
docker run -v $(pwd):/home/usampling-exp:z macher/usampling /bin/bash -c 'cd /home/usampling-exp/; echo STARTING; python3 usampling-experiments.py -flas /home/samplingfm/Benchmarks/Blasted_Real/blasted_case141.cnf /home/samplingfm/Benchmarks/Blasted_Real/blasted_case142.cnf --spur -t 1; echo END'
is calling SPUR sampler, with a timeout of 1 second, and with formulas explicitly given (here two formulas: useful to focus on specific formulas). You can also specify a folder.
Without flas
default formulas contained in the Docker folder/subfolders /home/samplingfm/
are processed (around 500 files).
We assess uniformity in two ways:
python3 barbarikloop.py -flas gilles --sampler 10 --seed 1 --timeout 60
where sampler is the sampler to be assessed (1=Unigen, 2=QuickSampler, 3=STS, 4=CMS, 5=UniGen3, 6=SPUR, 7=SMARCH, 8=UniGen2,9=KUS, 10=Distance-based Sampling), seed an integer seed and a timeout in seconds. it supports all the parameters of barbarik (use --help to see a description of all the options). We can also specify the sampler used as reference the same way with --ref-sampler
followed by the sampler to use.docker run -v $(pwd):/home/usampling-exp:z macher/usampling /bin/bash -c 'cd /home/usampling-exp/; echo STARTING; python3 barbarikloop.py -flas /home/samplingfm/Benchmarks/FeatureModels/FM-3.6.1-refined.cnf --sampler 9 --seed 1 --timeout 100; echo END'
(QuickSampler with JHipster feature model and timeout=100 seconds... end eta/epsilon defaut values cmd: ['python3', 'barbarik.py', '--seed', '1', '--verb', '1', '--eta', '0.9', '--epsilon', '0.3', '--delta', '0.05', '--reverse', '0', '--exp', '1', '--minSamples', '0', '--maxSamples', '9223372036854775807', '--sampler', '9', '--ref-sampler','6', '/home/samplingfm/Benchmarks/FeatureModels/FM-3.6.1-refined.cnf']
samplers
directory (and all utilities/dependencies are also in this folder)usampling-experiments.py
pilots the scalability study of samplers over different datasets barbarik.py
pilots the uniformity checking of samplers over different datasets. It is based on the barbarik tool from Kuldeep Meel et al: https://github.com/meelgroup/barbarik. This version supports uniformity check for all the 10 solvers above and uses KUS as a reference uniform solver, if not specified in the command line above.barbarikloop.py
allows to run uniformity checks on set fo files (using the same flas technique as above) and report the results in a CSV filecomputeDeviations.py
computes feature deviation graphs as proposed in the paper: "Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?" by Plazar et al., ICST 2019 (https://hal.inria.fr/hal-01991857). Under re-construction to support all solvers and improve usability.