SWIFTSIM / HBTplus

HBTplus halo finder adapted for the FLAMINGO and COLIBRE simulations
0 stars 0 forks source link

Try to make random sampling for unbinding reproducible #16

Open jchelly opened 4 months ago

jchelly commented 4 months ago

This makes the seed for the random shuffle used in unbinding depend on the particle IDs so that the random number sequence no longer depends on the (unpredictable) order in which threads process halos. It also adds a parameter that can be used to change the seed so that we can run multiple realizations. Particle IDs are xor'd with the supplied parameter and then used to initialize the random number generator. I'm not sure if some other operation would make more sense.

I think this change also fixes two problems with the original code:

There might still be some non-deterministic behaviour due to rounding error in reduction operations depending on the order of execution.

jchelly commented 4 months ago

This still doesn't seem to be enough to make runs deterministic with >1 threads (tested by running on L1000N0900/DMO_FIDUCIAL twice.)

jchelly commented 2 months ago

It looks like TrackIds somehow get assigned in a non-deterministic way when we use multiple threads. In the small Colibre test I get the same halo propertes in snapshot 000 but the TrackIds differ.

jchelly commented 2 months ago

FeedCentrals() assigns subhalo indexes in a non-deterministic way using a critical section. It could be fixed using an ordered section or just removing the openmp directives.

jchelly commented 2 months ago

With the ordered section fix two colibre test runs now generate identical output for snasphots 0-10 (of 15) but diverge after that.