set_loky_pickler issues with dill - Githubissues

eckelsjd / amisc

Efficient framework for building surrogates of multidisciplinary systems using the adaptive multi-index stochastic collocation (AMISC) technique.

https://eckelsjd.github.io/amisc/

GNU General Public License v3.0

1 stars 1 forks source link

set_loky_pickler issues with dill #20

Open eckelsjd opened 1 month ago

eckelsjd commented 1 month ago

Expected Behavior

dill is set as the pickler for joblib.Parallel in the ComponentSurrogate.refine() function to allow passing self objects to the workers
This should not affect anything about the component models that are run during the refine step

Actual Behavior

set_loky_pickler is global for joblib, so if the component model uses joblib, it will no longer be set to the default cloudpickle. This would force people's component models to switch joblib back to the appropriate pickler for their use case, which is undesirable.

Ideas

Can probably temporarily set the pickler to dill before calling any user code, then switch it back to default
Maybe there is a better way to do the parallel call in refine() in the first place to avoid the set_loky_pickler call altogether.

eckelsjd commented 1 month ago

On a related note, the component models cannot use joblib.Parallel at all. If they close down the Parallel object, then the refine() loop's Parallel object will stop working.