search

aig-upf / fs-private

This is the private version of the FS planner repository
GNU General Public License v3.0
5 stars 1 forks source link

Squash non-determinism #45

Open gfrances opened 7 years ago

gfrances commented 7 years ago

There seems to be some source of non-determinism in the C++ backend of the planner, which provokes some slight variation of results. This can be reproduced with the following command:

python3 preprocessor/runner.py --tag t4 --instance /home/gfrances/projects/code/fs-benchmarks/benchmarks/blocksworld-fn/instance_20_2.pddl --driver lazybfws --options "successor_generation=naive,bfws.rs=sim,evaluator_t=adaptive" --run

The json files generated by the frontend are identical, but number of expansions, etc. seem to be different. The logs generated by the (release-mode) planner are also equivalent.

miquelramirez commented 7 years ago

Is this confined to the functional domains? If so, surely is having an impact on the variance that I am seeing when using features.

gfrances commented 7 years ago

It seems so, although I'm not exactly sure what's going on... I've just found very small variations in the number of expanded nodes when running twice in a row the same command... I've only tried with the FN version of BW though...

miquelramirez commented 7 years ago

Ugh, I had forgotten about this. Today I sat for a long time with the proc. domains studying the effect of using features etc and observed wild variations in some domains like helper. Wild as in a 100% variation sometimes.

This variance happens even without features enabled, always when using BFWS (f7). For Blind BFWS, nothing out of place was observed after lots of runs done manually... Right now I am trying to do a controlled experiment on my laptop, to determine how big is the effect.