Menge has been designed to create probabilistically distributed simulations. A further effort has been made to make single samples from distributions repeatable (e.g., by passing a fixed value for the random seed). While a good faith effort, the implementation is incomplete and, in many practical, important ways, fails.
Distribution of initial state is correlated with distributions in simulation updates.
We have a single seed. That seed feeds all random number generators. That means, perturbing initial state implies perturbing the probabilistic components of the BFSM. It would be better if the two seeds could be separated (optionally). This would allow running a fixed population through varying behavior samples, or varying populations through a fixed behavior sample.
Non-determinism for fixed seeds based on OS-dependent thread artifacts.
While a concerted effort has been made to make the evaluation of the random number generators deterministic, it relies on deterministic code execution. I.e., if two agents, in a common state, both need to evaluate a random value from a shared generator, it will only be deterministic if the two agents are evaluated in a fixed order. However, using OpenMP means that the two agents could be evaluated in separate threads and the order of execution now becomes subject to the whims of the operating system. So, while the number generator passes out a deterministic sequence of pseudo-random values, the assignment and interpretation of those values is non-deterministic and leads to non-deterministic simulations.
There are a couple of actions that need to be taken to address this:
(#153) Running the simulation with a single thread will guarantee determinism. Currently, there is no way to accomplish that short of a) hacking the code, or b) running a debug build. We're missing a mechanism to declare the maximum number of computation threads at runtime.
(#154) Running in a single thread incurs a non-trivial performance cost (particularly for large simulations). It would be better if the mechanism addressed determinism even when evaluated across multiple threads.
We'll track efforts/designs to address these issues in sub-issues.
The problem
Menge has been designed to create probabilistically distributed simulations. A further effort has been made to make single samples from distributions repeatable (e.g., by passing a fixed value for the random seed). While a good faith effort, the implementation is incomplete and, in many practical, important ways, fails.
We'll track efforts/designs to address these issues in sub-issues.