search

cvxgrp / pymde

Minimum-distortion embedding with PyTorch
https://pymde.org
Apache License 2.0
538 stars 27 forks source link

Conflict with sklearn.manifold.TSNE #66

Closed kevinsweeney84 closed 1 year ago

kevinsweeney84 commented 2 years ago

Running in Jupyter notebook. Importing TSNE (from sklearn.manifold import TSNE) prior to pyMDE (import pymde) is causing the kernel to die when I try to use pyMDE (for example (mnist = pymde.datasets.MNIST())

The same is true in reverse, i.e. TSNE also kills kernel if the pyMDE is imported before TSNE

from sklearn.manifold import TSNE
import pymde

mnist = pymde.datasets.MNIST()
embedding = pymde.preserve_neighbors(mnist.data).embed()

Library versions: pymde - 0.1.15 scikit-learn - 1.1.1

akshayka commented 2 years ago

@kevinsweeney84, thanks for reporting. That's perplexing behavior.

Somehow, I was unable to reproduce this issue (I am running on Ubuntu 22.04).

Is there anything else you can tell me about your set-up? What operating system are you using, and how much RAM do you have?

kevinsweeney84 commented 2 years ago

Hello @akshayka. I am running on MacOS. I managed to "fix" the issue by installing nomkl (https://anaconda.org/conda-forge/nomkl) prior to installing sklearn or pymde. There seems to have been some conflict regarding the MKL used across both packages.

When running the code through terminal I got an error similar to 

OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. 
That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do 
is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking
 of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can 
set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, 
but that may cause crashes or silently produce incorrect results. For more information, please see 
[Intel® Product Support](http://www.intel.com/software/products/support/)

See https://discuss.pytorch.org/t/is-nomkl-still-recommended-for-omp-issue-on-mac/157388 as example of how I found fix. Thanks