[Bug]: interaction between random seed and readparfile

Describe the bug

During a jitter analysis (Marta Cousido) observed the following behaviour. When setting the seed in the starter file and using the "Initial Parameter Values=0" option, the seed seems to work correctly as I ran three models with the same seed and all three gave identical estimates.However, when I switch to "Initial Parameter Values=1" and set the seed in the starter file, the seed does not seem to work consistently. I ran three models with the same seed, but each gave different results. Is there any explanation for this behaviour?

To Reproduce

Rick will create a test example

Expected behavior

random seed to work as expected

Screenshots

No response

Which OS are you seeing the problem on?

No response

Which version of SS3 are you seeing the problem on?

No response

Additional Context

No response

I tried jittering without reading from the .par file and got different results each time.

It looks like the optional user-input random seed is used in the creation of bootstrap data files here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_write_ssnew.tpl#L22-L25 but not in the calculation of jittered initial values here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_objfunc.tpl#L1488.

If we were to use the user-controlled random seed for the jittered parameters then you would need to change the seed in the starter file each time your ran a new jitter. That's possible but might require some change in workflow. It might be tricky to use random seeds in the r4ss::jitter() function which now has parallelization, so I think the easier solution would just be to add an error if the user inputs a random seed. The benefit would be more reproducibility in the jittering process. Alternatively we could just document the fact that the seed only impacts the bootstrap data files, not the jitters.

@MartaCousido, can you tell us more about the use case you have and how important setting the seed in the jitters would be?

Thank you for taking the time to review this issue.

Perhaps I didn’t explain my use case clearly enough in my previous message. I am not using the r4ss::jitter() function to perform jittering. Instead, I am running the model multiple times, directly changing the seed in starter.ss for each run.

The problem I’m experiencing is that, when I run the jitter in this way, it seems that the model is not using the seed specified in starter.ss correctly (I provide a different seed for each jittered model). I suspect this because all the models return the same result, which is highly unusual for a model as unstable as mine. This makes me think that it’s pulling the seed from the system clock, and since all the models launch at the same time, they end up with identical results.

To confirm this theory, I ran two tests:

Instead of specifying a different seed for each jittered model in starter.ss, I left the seed field with -1 (no seed provided) and ran each model at a different time. As I expected, of the 30 models, only 8 had the same likelihood, indicating that something unusual happens when a fixed seed is set in starter.ss.

I conducted a separate test from the jittering process: I ran three models with the same seed in starter.ss and checked the results. The three models, despite having the same seed, gave different results, which again suggests that the model is using the system clock. Since these runs were started at different times, they produced different outputs.

The seed is important to us because using the clock as the seed source creates issues when we rely on a supercomputer with a queue system. In this environment, even if models are launched at different times, they can begin running simultaneously, which could result in identical outputs—not due to model stability, but because they’ve used the same seed.

Thank you again for looking into this, and please let me know if further clarification would be helpful.

Best regards, Marta Cousido

El jue, 14 nov 2024 a las 1:10, Ian Taylor @.***>) escribió:

I tried jittering without reading from the .par file and got different results each time.

It looks like the optional user-input random seed is used in the creation of bootstrap data files here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_write_ssnew.tpl#L22-L25 but not in the calculation of jittered initial values here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_objfunc.tpl#L1488 .

If we were to use the user-controlled random seed for the jittered parameters then you would need to change the seed in the starter file each time your ran a new jitter. That's possible but might require some change in workflow. It might be tricky to use random seeds in the r4ss::jitter() function which now has parallelization, so I think the easier solution would just be to add an error if the user inputs a random seed. The benefit would be more reproducibility in the jittering process. Alternatively we could just document the fact that the seed only impacts the bootstrap data files, not the jitters.

@MartaCousido https://github.com/MartaCousido, can you tell us more about the use case you have and how important setting the seed in the jitters would be?

— Reply to this email directly, view it on GitHub https://github.com/nmfs-ost/ss3-source-code/issues/643#issuecomment-2475078927, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANOTIQYOCTKFYWHCXYPFYTD2APTATAVCNFSM6AAAAABRU32LDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZVGA3TQOJSG4 . You are receiving this because you were mentioned.Message ID: @.***>

Thanks for the additional details @MartaCousido. Your findings support my conclusion that the jitters is SS3 are using the start time, not the random seed. But changing the code to make use of the random seed could impact how users typically run jitters.

@okenk and I thought about the possibility of simultaneous start times and matching seeds when jittering and thought that the "likelihood of the cores using the exact same seed is infinitesimal" https://r4ss.github.io/r4ss/reference/jitter.html#details. However, I now see that the C++ time() function used to set the seeds for the jitters only provides the time in seconds, so simultaneous start times could easily occur: https://en.cppreference.com/w/c/chrono/time. One alternative solution might be to use a more precise clock using something like https://en.cppreference.com/w/cpp/chrono/high_resolution_clock. Additionally the r4ss::jitter() function (if you're using it) could build in some kind of random delay to the starting time to avoid simultaneous starts.

Hi Ian,

Thank you for looking into this issue and suggesting possible solutions.

As you point out, the probability of two models starting at exactly the same time is indeed high. As mentioned in my previous email, this has happened to us many times. We raised this issue to explore the possibility of specifying a different seed for each jittered model, independent of the start time. However, if this isn't possible, we have implemented a workaround in our code. Specifically, we've adapted the submission process to the supercomputer's queuing system so that models are started at staggered times. Occasionally, some models will still enter at the same time if they are held in the queue, but in these cases we remove the duplicate model from the jitter results.

Thanks again for your time and effort in the investigation of this issue.

El vie, 15 nov 2024 a las 17:52, Ian Taylor @.***>) escribió:

Thanks for the additional details @MartaCousido https://github.com/MartaCousido. Your findings support my conclusion that the jitters is SS3 are using the start time, not the random seed. But changing the code to make use of the random seed could impact how users typically run jitters.

@okenk https://github.com/okenk and I thought about the possibility of simultaneous start times and matching seeds when jittering and thought that the "likelihood of the cores using the exact same seed is infinitesimal" https://r4ss.github.io/r4ss/reference/jitter.html#details. However, I now see that the C++ time() function used to set the seeds for the jitters only provides the time in seconds, so simultaneous start times could easily occur: https://en.cppreference.com/w/c/chrono/time. One alternative solution might be to use a more precise clock using something like https://en.cppreference.com/w/cpp/chrono/high_resolution_clock. Additionally the r4ss::jitter() function (if you're using it) could build in some kind of random delay to the starting time to avoid simultaneous starts.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-ost/ss3-source-code/issues/643#issuecomment-2479425005, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANOTIQ6Q3KW6ROEYDGJVUU32AYRGLAVCNFSM6AAAAABRU32LDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZZGQZDKMBQGU . You are receiving this because you were mentioned.Message ID: @.***>

nmfs-ost / ss3-source-code