dill is set as the pickler for joblib.Parallel in the ComponentSurrogate.refine() function to allow passing self objects to the workers
This should not affect anything about the component models that are run during the refine step
Actual Behavior
set_loky_pickler is global for joblib, so if the component model uses joblib, it will no longer be set to the default cloudpickle. This would force people's component models to switch joblib back to the appropriate pickler for their use case, which is undesirable.
Ideas
Can probably temporarily set the pickler to dill before calling any user code, then switch it back to default
Maybe there is a better way to do the parallel call in refine() in the first place to avoid the set_loky_pickler call altogether.
On a related note, the component models cannot use joblib.Parallel at all. If they close down the Parallel object, then the refine() loop's Parallel object will stop working.
Expected Behavior
dill
is set as the pickler forjoblib.Parallel
in theComponentSurrogate.refine()
function to allow passingself
objects to the workersActual Behavior
set_loky_pickler
is global for joblib, so if the component model uses joblib, it will no longer be set to the defaultcloudpickle
. This would force people's component models to switch joblib back to the appropriate pickler for their use case, which is undesirable.Ideas
dill
before calling any user code, then switch it back to defaultrefine()
in the first place to avoid theset_loky_pickler
call altogether.