Closed AngelFP closed 2 months ago
With regards to my first comment, I wonder if it would be a good idea to backup automatically the history file from which the exploration run is resuming. Previous versions had a feature like this if I recall well.
About the other point, I just added a commit where the same strategy for considering (or not) trials with parameters outside the design range is replicated in AxModelManager
.
A unit test is failing here:
FAILED tests/test_ax_generators.py::test_ax_single_fidelity_resume - AssertionError: assert 'Sobol' == 'GPEI'
This happens when the exploration resumes with fit_out_of_design=True
.
It is expected that, when trials with parameters outside the design range are added, the next generated point uses the model. However, the generation method is still 'Sobol' in this case.
Could the expected behavior be in conflict with the changes implemented here? -> https://github.com/optimas-org/optimas/pull/207
I have made additional checks and it seems that this issue is always present when one resumes an optimas run. So it is nothing particular to this PR, but rather to this -> https://github.com/optimas-org/optimas/pull/207 or even from before. However only the new unit tests associated to this PR are failing, because it is the only one checking the generation method of the new trials after resuming.
With regards to my first comment, I wonder if it would be a good idea to backup automatically the history file from which the exploration run is resuming. Previous versions had a feature like this if I recall well.
About the other point, I just added a commit where the same strategy for considering (or not) trials with parameters outside the design range is replicated in
AxModelManager
.
Thanks for adding fit_out_of_design
to the AxModelManager
. I'm not a fan of how we currently have to duplicate some code in the generator and the AxModelManager
, so we should find a way of unifying this in a future PR.
I will have a look at the other issues.
A unit test is failing here:
FAILED tests/test_ax_generators.py::test_ax_single_fidelity_resume - AssertionError: assert 'Sobol' == 'GPEI'
This happens when the exploration resumes with
fit_out_of_design=True
. It is expected that, when trials with parameters outside the design range are added, the next generated point uses the model. However, the generation method is still 'Sobol' in this case.Could the expected behavior be in conflict with the changes implemented here? -> #207
Ok, this issue should be fixed now.
With regards to my first comment, I wonder if it would be a good idea to backup automatically the history file from which the exploration run is resuming. Previous versions had a feature like this if I recall well.
I think that saving a backup (or just using a different name for the history file after resume) could indeed be a good idea. Otherwise it will always be overwritten and can lead to issues like the one you describe. This should be the focus of a future PR.
This enables, among others, the possibility of resuming an old exploration with an updated range of the varying parameters.
Previously, trying to add trials outside of the current design space when using and
AxServiceGenerator
would fail because the AxClient would not allow it. In this PR, this workaround is used whenfit_out_of_design=True
so that trials outside of the current range will still contribute to the surrogate model.When
fit_out_of_design=False
(default), the out-of-range trials will be ignored by the generator and the user will be informed about it.