`choose_generation_strategy` not respecting parallelism override + trying to use winsorization when it (maybe?) shouldn't

[aaditya8959]

I have set up an optimization experiment with 17 input parameters and 2 objectives. Before iterating and obtaining trial suggestions, I initialized the client with an initial dataset containing a mixture of trials with objective values, and trials without objective values(failed observations which I log as a trial failure in the client). As the client runs and produces more trials some of the inputs result in objective values while some are logged as trial failure, and this is purely based on constraints on the inputs that I evaluate. Additionally, I am querying multiple trials in an iteration, for which I have set the following while defining the client :

choose_generation_strategy_kwargs = {'max_parallelism_override':max_num_suggestions_per_iter}

and during the iterations I query multiple trials via

parameters, trial_index = ax_client.get_next_trials(max_num_suggestions_per_iter)

As I run the iterations I noticed the following :

I have set max_num_suggestions_per_iter to 50. Even with the max_parallelism_override set to 50, the client is able to generate not more than 34 trials per iteration. Is that a hard limit on the number of trials irrespective of the override set or am I missing something?

After a few iterations I see the following message :

**/home/hegadekattev/python_projects/project_env_basic/lib/python3.8/site-packages/ax/modelbridge/transforms/winsorize.py:275: UserWarning:

Automatic winsorization isn't supported for an objective in MultiObjective without objective thresholds. Specify the winsorization settings manually if you want to winsorize metric BucklePressure.

/home/hegadekattev/python_projects/project_env_basic/lib/python3.8/site-packages/ax/modelbridge/transforms/winsorize.py:275: UserWarning:

Automatic winsorization isn't supported for an objective in MultiObjective without objective thresholds. Specify the winsorization settings manually if you want to winsorize metric ShellWeight.

[INFO 07-05 08:54:40] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.

[INFO 07-05 08:54:40] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.**

and no more trials are suggested and the loop seems in a standstill. Can you please elaborate on the warning message and the reason that might be happening?

Thank you, Aadi

[lena-kashtelyan]

Hi @aaditya8959, apologies for the delay on this one!

Could you please provide:

1. Full code snippet you are running (will need to see all calls to AxClient methods to understand what is going on), 2. Output of AxClient.experiment_data_frame, to show us the trials you are getting?

See next comment.

[lena-kashtelyan]

Actually, I think I might know what is going on here. Your parallelism override will take affect, but only after the first batch. This is because:

The first batch of trials is generated with Sobol, and number of trials in it is auto-chosen by this function: https://github.com/facebook/Ax/blob/2db0f2abc21d959e2ac18e3fa0f3c3424f689e6d/ax/modelbridge/dispatch_utils.py#L257. Clearly this logic is not working too well in this case, and we could consider automatically setting the number of Sobol trials to the maximum parallelism level. However, Sobol trials are chosen quasi-randomly, so this could mean running more quasi-random trials and wasting compute and time, instead of running just enough and switching to Bayesian optimization as soon as possible (after 17 * 2 = 34 trials in this case).

If you pass choose_generation_strategy_kwargs={"num_initialization_trials": max_num_suggestions_per_iter, 'max_parallelism_override': max_num_suggestions_per_iter}, things should work as you expect.

The winsorization thing is actually unrelated and is not an issue. It's just telling you and we can't automatically perform winsorization in multi-objective settings; it must be configured manually. But it's not essential and ok to skip; I'll see if we can / should silence this warning.

Also, you might be interested in using our SAASBO model, which performs better in higher-dimensional search spaces. To use it, you can also add a third argument to your choose_generation_strategy_kwargs, like so: choose_generation_strategy_kwargs={"num_initialization_trials": max_num_suggestions_per_iter, 'max_parallelism_override': max_num_suggestions_per_iter, "use_saasbo": True}. Note that SAASBO is a bit slower than our other models though, and generating 50 trials might take a while. All available kwargs you can pass are listed here by the way: https://github.com/facebook/Ax/blob/2db0f2abc21d959e2ac18e3fa0f3c3424f689e6d/ax/modelbridge/dispatch_utils.py#L278; docstring of this function will be informative for understanding what the different arguments are doing.

Finally, my question to you: why are you setting the parallelism to such a high level (50)? How many trials are you expecting to run total? Ax provides algorithms for sample-efficient optimization, and 50 might be higher parallelism than you need.

[lena-kashtelyan]

Hi @aaditya8959! Please let us know if the answer I gave above is not exhaustive/if there is still something you are wondering. Please reopen the issue if you follow up, as we are unlikely to see comments on a closed issue.