Closed markleader closed 3 years ago
Maybe we need to have a broader discussion about the correct responses for a distributed optimizer.
We have three quantities:
Design vars and constraints can either be local or global. I suppose in the general case of multi-objective problems it might be possible to have a distributed objective... but that seems weird to me. I think the objective(s) should be global always.
So that would imply that get_objective_values()
should
a) return the local copy of the global value
b) broadcast the value from the root proc to all procs and return that
We should choose one of these and go with it. There does not seem to be a need for optional selection of a/b here to me.
For get_constraint_values
and get_design_var_values
we need to distinguish what you want:
a) local + global (may or may not need a broadcast from root, depending on how we decide to handle this)
b) local + global + remote (requires an all gather)
It will maybe be important for the user to know which constarints/dvs are local/remote/global... so that metadata needs to be available too.
So in my mind the simplest case for a distributed optimization algorithm is this:
My view is that this should be the default, but that you could have options for a distributed constraint vector as well.
How the constraint values are broadcast or stored on each process isn't as much of an issue for the optimization algorithm itself, but is, of course, an issue for OpenMDAO.
The other issue here is the gradient/Jacobian, but in my view that should follow from the distribution of the design variables.
The other issue here is the gradient/Jacobian, but in my view that should follow from the distribution of the design variables ... and constraints
I agree with that philosophy. But let me ask a few follow up questions:
1) if you have a distributed design vector, should OpenMDAO ever be expected to gather it all to one proc (i.e. should we let a non-distributed optimizer work in this situation)? I would think the answer is yes, even if just so we can run comparisons. You really don't want to have to re-structure the problem to switch optimizers. I know that some problems will be far too big to do the gather with... but I think OpenMDAO should still have the ability to do it.
2) same question, but with constraints.
3) same question, but with jacobian.
If the answer in all cases is "the driver should be able to query either the local + global, or the local+global+remote. Its up to the driver to do it right" then that gives us good bounds for the API. The other option is "The driver has to respect the structure that the model uses. So it can really only every have local+global".
Im not a fan of the latter, but Im open to someone attempting to change my mind. The "local+global" only would provide a simpler API and be a little less work... but I don't think the restriction is worth it. Im pretty confident that users will try to swap out to a non parallel driver and be frustrated when they get an error saying "fix your model. this won't work".
I agree, I think the driver should have options to pick the data that it needs, with the associated penalties in computational time the further the desired problem structure deviates from the actual problem structure.
@markleader is this till an open issue?
@JustinSGray No, we're good here - closed it.
Summary of Issue
Allow constraints to be distributed for parallel optimization.
Issue Type
Description
Allow
driver.get_constraint_values
to (optionally) be a distributed operation.Mimic the API in
get_design_var_values
, add aget_remote
argument to theget_constraint_values
method in driver.When
get_remote=False
, then the constraint values should not be gathered. Whenget_remote=True
, then the constraint values will be gathered. (this should be the default)NOTE: Default behavior should remain unchanged.
Example
Run the example with mpirun -np 2 python.