search

dstl / Stone-Soup

A software project to provide the target tracking community with a framework for the development and testing of tracking algorithms.
https://stonesoup.rtfd.io
MIT License
400 stars 131 forks source link

Iterating over sets breaks reproducability of tutorials #718

Closed jvbrink closed 1 year ago

jvbrink commented 1 year ago

Some of the tutorials, for example Tutorial 6, uses sets and then iterates over these. As Python sets are unordered, the iteration order is arbitrary. There is no major issue with this, but as for instance random measurements are generated in an iterative manner with a seeded RNG - this can affect the result of a run, even with the RNG is seeded.

For instance, I am able to reproduce the results you show in Tutorial 6, but adding a simple breakpoint() to the code, which should not change behavior of the script, changes the iteration order over the set, and thus I get completely different results. Below are the two possible outputs (only difference being a breakpoint, or a non-consequential variable definition, e.g. a=0 at the bottom of the script). I am not exactly sure what decides the set iteration order, as it does seem to be static for a given code on re-runs (i.e., not time dependent).

result_variant_a result_variant_b

I suggest the tutorials that rely on arbitrary iteration order change these, so that they become reproduicble.

sdhiscocks commented 1 year ago

Thanks for raising this @jvbrink.

ordered_set.OrderedSet is already used in some of the other examples and simulators to ensure reproducible output, but is missing in some places where the order makes a difference, like your example.

It would be good to review where sets are being used, and when the order has an impact, replace with OrderedSet.