search

explosion / spaCy

💫 Industrial-strength Natural Language Processing (NLP) in Python
https://spacy.io
MIT License
30.19k stars 4.4k forks source link

Installing spacy[cuda] via pip on AWS SageMaker extremely slow (~1hr+) #5579

Closed mgbvox closed 4 years ago

mgbvox commented 4 years ago

How to reproduce the problem

I'm running a standard-issue AWS p2.xlarge SageMaker Jupyter Lab notebook instance. To replicate: either via in-lab terminal or from a notebook, run: source activate pytorch_p36 # this only on terminal !pip install -U spacy[cuda]

The installation proceeds and completes as necessary; however, the cupy wheels building step appears to hang for at least one hour (or at least, outputs no progress report beyond Building wheels for collected packages: cupy Running setup.py bdist_wheel for cupy ...). I haven't been able to stick around long enough to see it complete; typically, I go walk my dog, and by the time I return, I still have time to make lunch before the installation finishes. The process is similarly slow - and sticks in the same spot, e.g. cupy wheels building - on google colab.

I attempted to include a spacy[cuda] install in the Life Cycle config for my notebook, but then startup fails after a very long wait period.

No Error - just SLOW.

Your Environment

Info about spaCy

pip freeze:

alabaster==0.7.10 anaconda-client==1.6.14 anaconda-project==0.8.2 asn1crypto==0.24.0 astroid==1.6.3 astropy==3.0.2 attrs==18.1.0 autovizwidget==0.15.0 awscli==1.18.39 Babel==2.5.3 backcall==0.1.0 backports.shutil-get-terminal-size==1.0.0 bcrypt==3.1.7 beautifulsoup4==4.6.0 bitarray==0.8.1 bkcharts==0.2 blaze==0.11.3 bleach==2.1.3 blis==0.2.4 bokeh==1.0.4 boto==2.48.0 boto3==1.12.39 botocore==1.15.39 Bottleneck==1.2.1 cached-property==1.5.1 certifi==2019.11.28 cffi==1.11.5 chardet==3.0.4 click==6.7 cloudpickle==0.5.3 clyent==1.2.2 colorama==0.3.9 contextlib2==0.5.5 cryptography==2.9 cycler==0.10.0 cymem==2.0.2 Cython==0.28.2 cytoolz==0.9.0.1 dask==0.17.5 dataclasses==0.6 datashape==0.5.4 decorator==4.3.0 defusedxml==0.6.0 distributed==1.21.8 docker==4.2.0 docker-compose==1.25.5 dockerpty==0.4.1 docopt==0.6.2 docutils==0.14 entrypoints==0.2.3 enum34==1.1.9 environment-kernels==1.1.1 et-xmlfile==1.0.1 fastai==1.0.60 fastcache==1.0.2 fastprogress==0.2.2 filelock==3.0.4 Flask==1.0.2 Flask-Cors==3.0.4 gevent==1.3.0 glob2==0.6 gmpy2==2.0.8 greenlet==0.4.13 h5py==2.8.0 hdijupyterutils==0.15.0 heapdict==1.0.0 html5lib==1.0.1 idna==2.6 imageio==2.3.0 imagesize==1.0.0 importlib-metadata==1.5.0 ipykernel==4.8.2 ipyparallel==6.2.2 ipython==6.4.0 ipython-genutils==0.2.0 ipywidgets==7.4.0 isort==4.3.4 itsdangerous==0.24 jdcal==1.4 jedi==0.12.0 Jinja2==2.10 jmespath==0.9.4 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.2.3 jupyter-console==5.2.0 jupyter-core==4.4.0 jupyterlab==0.32.1 jupyterlab-launcher==0.10.5 kiwisolver==1.0.1 lazy-object-proxy==1.3.1 llvmlite==0.23.1 locket==0.2.0 lxml==4.2.1 MarkupSafe==1.0 matplotlib==3.0.3 mccabe==0.6.1 mistune==0.8.3 mkl-fft==1.0.6 mkl-random==1.0.1 mock==4.0.1 more-itertools==4.1.0 mpmath==1.0.0 msgpack==0.6.0 msgpack-python==0.5.6 multipledispatch==0.5.0 murmurhash==1.0.2 nb-conda==2.2.1 nb-conda-kernels==2.2.2 nbconvert==5.4.1 nbformat==4.4.0 networkx==2.1 nltk==3.3 nose==1.3.7 notebook==5.5.0 numba==0.38.0 numexpr==2.6.5 numpy==1.15.4 numpydoc==0.8.0 odo==0.5.1 olefile==0.45.1 onnx==1.5.0 opencv-python==4.1.1.26 openpyxl==2.5.3 packaging==20.1 pandas==0.24.2 pandocfilters==1.4.2 paramiko==2.7.1 parso==0.2.0 partd==0.3.8 path.py==11.0.1 pathlib2==2.3.2 patsy==0.5.0 pep8==1.7.1 pexpect==4.5.0 pickleshare==0.7.4 Pillow==5.4.1 pkginfo==1.4.2 plac==0.9.6 plotly==4.5.2 pluggy==0.6.0 ply==3.11 preshed==2.0.1 prompt-toolkit==1.0.15 protobuf==3.5.2 protobuf3-to-dict==0.1.5 psutil==5.4.5 psycopg2==2.7.5 ptyprocess==0.5.2 py==1.5.3 py4j==0.10.7 pyasn1==0.4.8 pycodestyle==2.4.0 pycosat==0.6.3 pycparser==2.18 pycrypto==2.6.1 pycurl==7.43.0.1 pyflakes==1.6.0 pygal==2.4.0 Pygments==2.2.0 pykerberos==1.2.1 pylint==1.8.4 PyNaCl==1.3.0 pyodbc==4.0.23 pyOpenSSL==18.0.0 pyparsing==2.2.0 PySocks==1.6.8 pyspark==2.3.4 pytest==3.5.1 pytest-arraydiff==0.2 pytest-astropy==0.3.0 pytest-doctestplus==0.1.3 pytest-openfiles==0.3.0 pytest-remotedata==0.2.1 python-dateutil==2.7.3 pytz==2018.4 PyWavelets==0.5.2 PyYAML==5.3.1 pyzmq==17.0.0 QtAwesome==0.4.4 qtconsole==4.3.1 QtPy==1.4.1 requests==2.20.0 requests-kerberos==0.12.0 retrying==1.3.3 rope==0.10.7 rsa==3.4.2 ruamel-yaml==0.15.35 s3fs==0.1.5 s3transfer==0.3.3 sagemaker==1.55.3 sagemaker-pyspark==1.3.0 scikit-image==0.13.1 scikit-learn==0.20.3 scipy==1.1.0 seaborn==0.8.1 Send2Trash==1.5.0 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.11.0 smdebug-rulesconfig==0.1.2 snowballstemmer==1.2.1 sortedcollections==0.6.1 sortedcontainers==1.5.10 spacy==2.1.8 sparkmagic==0.12.5 Sphinx==1.7.4 sphinxcontrib-websupport==1.0.1 spyder==3.2.8 SQLAlchemy==1.2.11 srsly==0.1.0 statsmodels==0.9.0 sympy==1.1.1 tables==3.4.3 TBB==0.1 tblib==1.3.2 terminado==0.8.1 testpath==0.3.1 texttable==1.6.2 thinc==7.0.8 toolz==0.9.0 torch==1.4.0 torchvision==0.5.0 tornado==5.0.2 tqdm==4.42.1 traitlets==4.3.2 typing==3.6.4 typing-extensions==3.7.4.1 unicodecsv==0.14.1 urllib3==1.23 wasabi==0.2.2 wcwidth==0.1.7 webencodings==0.5.1 websocket-client==0.57.0 Werkzeug==0.14.1 widgetsnbextension==3.4.2 wrapt==1.10.11 xlrd==1.1.0 XlsxWriter==1.0.4 xlwt==1.3.0 zict==0.1.3 zipp==3.0.0

Conda YAML (exported after install):

name: pytorch_p36 channels:

mgbvox commented 4 years ago

NOTE:

nvcc --version gives 10.0. However, installing spacy[cuda100] and then: import spacy followed by spacy.require_gpu() throws: ValueError: GPU is not accessible. Was the library installed correctly?

mgbvox commented 4 years ago

Further NOTE:

while nvcc --version does give 10.0, import torch followed by torch.version.cuda() gives 10.1. Attempting pip install -U spacy[cuda101]gives the does not provide the extra 'cuda101' error.

adrianeboyd commented 4 years ago

I think you need to figure out which CUDA version is correct for your environment, since I'm not sure where the conflicting information is coming from. cupy doesn't depend on torch, so most likely nvcc --version is what you should rely on, but I'm not 100% sure about what's going on in your environment here.

But once you have the right CUDA version, it's usually easiest just to install the corresponding binary package for cupy by hand. The order of the installs doesn't matter, so you can install it before or after installing spacy and it'll be detected correctly when you enable the GPU in spacy.

pip install spacy
pip install cupy-cuda100

This should be equivalent to pip install spacy[cuda100], but it might be easiest to figure out what's going on by running them separately.

It looks like the cupy package installs from source, which might be what is taking so long. I'd recommend the binary cupy-cuda100 packages instead. Make sure that you clean up any lingering cupy packages in advance, since they conflict.

mgbvox commented 4 years ago

Thanks for your detailed response!

How would one get the cupy binaries? Is there a way to specify binaries using pip? Or would I need to download from the cupy website?

adrianeboyd commented 4 years ago

cupy is the source package, cupy-cudaNNN are the binary packages.

github-actions[bot] commented 4 years ago

This issue has been automatically closed because it was answered and there was no follow-up discussion.

github-actions[bot] commented 3 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.