Open nbeuchat opened 3 years ago
If anybody has the same problem, you can use the following workaround:
ColumnTransformer
with little data so that it doesn't crashThis requires the pipeline to be broken up into different parts that can be trained separately but that does the trick.
# First fit the colum transformers with little data
dimensionality_reduction.fit(X[:1000, :], y[:1000])
# Then fit the UMAP selector again with the whole dataset
dimensionality_reduction.named_transformers_["umap_selector"].fit(X, y)
I'm not really quite sure what is going astray here, because, as you note, it works fine outside of the pipeline. My best guess just looking at the code used and the error produced is that there is some catch with the slice(-512, None)
resulting in something being passed in to UMAP that isn't a simple numpy array (lazy evaluation perhaps?). One option might be to make an explicit list of columns (using np.arange
or list(range(.)
).
I'll see if I can reproduce this and dig a little deeper into exactly what aspect of this is unpickleable.
Thanks for your answer! The unexpected thing is that it works for smaller datasets when I do dimensionality_reduction.fit(X[:1000, :], y[:1000])
Let me know if you want me to test anything!
That's not actually unexpected. For small enough dataset sizes (the somewhat arbitrary threshold is 4096 samples) the fancy NNDescent nearest neighbor search is overkill, and it is cheaper to just compute all-pairs distances. That avoids going through the routine that is currently failing, so anything below 4096 samples should work.
Got it, interesting, thanks for the info!
In good news I can reproduce the error. In bad news I am even more puzzled as to what is going on -- The objects that UMAP is getting should be fine.
As far as I can tell it seems to be related to numba/numba#3370 , but seems related to the double nesting of joblib Parallel calls. I'll see if I can get any advice on this from the numba team.
It looks like it is the fact that ColumnTransformer wraps everything in a joblib.Parallel
call and, even if n_jobs=1
that changes the parallel context, and thus induces pickling, which doesn't play well with numba recursive functions. Normally this is all fine because the internal joblib.Parallel
call inside pynndescent has an explicit prefer="threads"
which uses the threading backend and avoid the serialization issues. I think (but still have to verify) that the outer Parallel call is messing that up. As a workaround for now I believe you should be able to do:
from joblib import parallel_backend
n_umap_components = 10
umap_reduction = Pipeline([
("sent_vect_umap", UMAP(n_components = n_umap_components)),
("umap_scaler", StandardScaler())
])
dimensionality_reduction = ColumnTransformer([
("umap_selector", umap_reduction, slice(-512, None)),
], remainder="passthrough")
with parallel_backend('threading'):
dimensionality_reduction.fit_transform(X, y)
which should enforce the threading backend, and make everything play nice. Of course this is bad if you want actual parallelism in your ColumnTransformer, since parallelisation there may well run afoul of the GIL, but as long as you only need that to run serially (e.g. n_jobs=1
) it should be fine.
That's awesome, thanks so much for investigating this that quickly! I'll try your proposed approach, I'll write back how it goes.
You may want to assign the result of the fit transform to something -- otherwise it'll get lost in the with context I realised. I think I have a fix for this now to go into pynndescent that will make everything good again: rather than prefer="threads"
I need to use require="sharedmem"
to stop the lack of serialization getting overridden from outside.
Thanks for reporting this -- it has certainly been a fun (if occasionally maddening) plunge into a a rabbit hole of complex interactions between different dependencies (joblib, numba).
Your trick worked like a charm!
FWIW, it's only needed for the fit
/fit_transform
which is great to use the component in production.
Thanks for reporting this -- it has certainly been a fun (if occasionally maddening) plunge into a a rabbit hole of complex interactions between different dependencies (joblib, numba).
I can only imagine the ramifications ;-)
Serialization support for _rp_forest
would also probably fix this – discussion for that is #273 and lmcinnes/pynndescent#93.
(Assuming one really wanted to serialize the whole UMAP object and send it to every process. More important for deserializing from storage for production use.)
I have a scikit-learn pipeline that uses
UMAP
for dimensionality reduction. It works fine for a small dataset (I tried 2k samples with 512 dimensions, 10 UMAP components) but when I use a larger dataset (50k samples, 512 dimensions, 10 UMAP components), it always fails when fitting with the following error:PicklingError: Could not pickle the task to send it to the workers
(full error below)The relevant part of the pipeline is simply (which is fed into a larger pipeline):
If I do:
dimensionality_reduction.fit_transform(X,y)
, it fails with the error mentioned above. But if I do:umap_reduction.fit_transform(X,y)
, it works fine.I believe that this is due to the parallelization of ColumnTransformer (it would be the same as FeatureUnion) but in my case,
n_jobs
isNone
by default and therefore uses a single process.I have the following packages versions:
I am running this in Jupyter Lab. Note that when I trained other UMAP models without a scikit-learn pipeline, it worked fine even with much larger amount of data.
Given that UMAP is by far the dimensionality reduction method that gave me the best results on my dataset, I'd really like to be able to include it in our pipeline! Any help would be very much appreciated.
Thanks a lot for your help!! Nicolas
Full error log: