bark-simulator / bark

Open-Source Framework for Development, Simulation and Benchmarking of Behavior Planning Algorithms for Autonomous Driving
https://bark-simulator.github.io/
MIT License
288 stars 69 forks source link

Repeatability? #569

Closed mrzheng8 closed 2 years ago

mrzheng8 commented 2 years ago

Repeatability is mentioned In the docement "BARK: Open Behavior Benchmarking in Multi-Agent Environments". “Experiments in BARK shall be fully reproducible, independent of the frequency at which the simulation runs.” However, when I run the example by run "bazel run //bark/examples:intersection", I find that it runs multiple times and the execution results are different. What causes this difference? Besides, I didn't find the bag bark.core. "from bark.core.models.behavior import *", I wonder how the core bag can be found. I would appreciate it if you could help me answer these two questions!

juloberno commented 2 years ago

Thanks for pointing this out. I checked it in the example.

The paper uses the ConfigurableScenarioGeneration, which allows setting a parameter for the random seed, which achieves reproducible scenario sets for a given parameter file. The ConfigWithEase scenario generation used in the examples does not include this option. A workaround is to add

numpy.random.seed(some_arbitrary_seed)

at the beginning of the example's python file.

To your second question: You find all files generated during the build in the bazel build file folders, e.g. for the intersection example you have ".../bark/bazel-bin/bark/examples/intersection.runfiles/bark_project/bark/core.so".