Use 'best' result during scenario evaluation

[tgeijten]

Use the best result (or first row in the .par file) when evaluating a scenario with Ctrl + E. This will allow proper playback of previous .par files in new scenarios.

Alternatively, add an option use_init_file_best to Optimizer, which causes the optimizer to use the 'best' result for the population mean instead of the mean.

[mitkof6]

@tgeijten, I am having this strange behavior, which I cannot explain. The same version of SCONE built on two different machines. I optimize on the first and then copy the result folder. If I evaluate the model (through the optimization window), it falls and gets a very different score from the one on the machine which was optimized.

This is annoying because I am optimizing on a cluster. However, when I check the results on my machine they give me different solutions. Should solutions be so sensitive to machine specifics?

[tgeijten]

@mitkof6, this indeed happens, unfortunately, and it's the result of hardware-specific differences in floating point calculations. Even though these differences are minuscule, errors accumulate in a predictive simulation, so a stable controller can become unstable on different hardware.

The 'good' news is that if you re-start an optimization using the result as input, it will quickly (after only a couple of iterations) become similar to the original.

One possible remedy is to increase the accuracy/strictness level of the floating point model during compilation, but this will be at the expense of simulation time. It's also quite a bit of work and is not guaranteed to solve the problem, unless your current accuracy level is low (not sure what the options are on Linux).

A workaround strategy is to make your controllers more robust against these small differences, for instance by adding a NoiseController. The outcomes will still be different, but at least they are much closer by default.

[mitkof6]

Thanks. Yes, I thought so. Indeed, I started adding noise to the optimization to test if this will improve results. The problem becomes more frequent once I increase the number of parameters in the model. Also, sometimes the solution could be on the verge of instability (without noise).

[mitkof6]

@tgeijten I am testing the following strategy: Evaluate the model multiple times and report the worst score. During the evaluation, I introduce random noise with a random seed so that each evaluation is unique. I will let you know if this works well and make a PR.

Another idea that came to mind is to store the .par files in a binary format and not text. Would this feature be possible to implement? If you do not have time, could you please suggest what things must be updated for this to work (e.g., store and evaluate binary par files)?

[tgeijten]

I’m not sure I understand the problem you’re trying to solve with your strategy, could you elaborate?

Also, why do you want to store the parameters in binary format?

[mitkof6]

Having random noise does not guarantee that a solution is robust. Because it can happen that for a particular noise sequence, we might be lucky and score well. To improve the robustness, we can re-evaluate our simulation multiple times for the same search point. Having random noise ensures that these multiple runs on the same machine produce different results. We can then return the objective value of the worst simulation. If luck led to a good score, then using multiple evaluations would reduce this effect and lead to more robust solutions. This is at least what I would expect, but I have to test this.

[mitkof6]

Do you think storing results in binary format would resolve the aggregation of numerical errors about the parameter files? I believe that in the text format, we are losing decimals points that might be important. I was looking for an easy way to test this, but I think it goes down to the submodules. Also, it would be nice to have an option to store the simulation .sto each time the best solution is obtained. In that way, we could at least preview the solutions on different machines.

[tgeijten]

Re random noise: I have found the best (and easiest) solution to this is to simply make the simulation time longer. Even if you run the simulation multiple times, you can still get "lucky" in all the runs. In fact, the chances of getting lucky in three 10 second simulations are similar to the chances of getting lucky in a single 30 second simulation. Also, choosing a non-deterministic random seed can be problematic for the optimizer and make it harder to reproduce your results. If you're worried that a controller will optimize for a specific random sequence (which is unlikely), you can choose to change the random seed based on a deterministic value (e.g. the lower decimals of the parameter values). However, I strongly doubt this is necessary, unless you're using something like a high-resolution feed-forward controller that can adapt to individual random samples (in which case, using direct collocation would be a much better approach).

Re binary format: there is no aggregation of numerical errors in the parameter file, so this is not an issue. The parameter values are already rounded before the optimizer evaluates them (see search_point::set_values()), so the parameters used during optimization are identical to those stored in the .par file. This way, on compatible hardware, the results will already exactly be the same during evaluation and optimization. The only issue is due to hardware-specific numerical differences during simulation.