Issue installing featuretools

[FaizSaeed]

I am trying to install featuretools using:

pip install featuretools

I already have almost all the packages installed that are commonly used for data analysis. However, when I try to run the pip command, I get this error:

Cannot uninstall 'cloudpickle': It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Please guide me as to how I can resolve this issue.

---

Issues created here on Github are for bugs or feature requests. For usage questions and questions about errors, please ask on Stack Overflow with the featuretools tag. Check the documentation for further guidance on where to ask your question.

[kmax12]

@FaizySaeed can you share the versions of your operating system, python installation, pip? Please also run pip freeze and share the output. Thanks!

[FaizSaeed]

Operating System: Windows 10 Version 1803 (OS Build 17134.165)

Pip Version: pip 18.0

Python Version: Python 3.6.1 :: Anaconda 4.4.0 (64-bit)

Pip Freeze Output: alabaster==0.7.10 anaconda-client==1.6.3 anaconda-navigator==1.6.2 anaconda-project==0.6.0 asn1crypto==0.22.0 astroid==1.4.9 astropy==1.3.2 Babel==2.4.0 backports.shutil-get-terminal-size==1.0.0 beautifulsoup4==4.6.0 bitarray==0.8.1 blaze==0.10.1 bleach==1.5.0 bokeh==0.12.5 boto==2.46.1 boto3==1.7.67 botocore==1.10.67 Bottleneck==1.2.1 cffi==1.10.0 chardet==3.0.3 click==6.7 cloudpickle==0.2.2 clyent==1.2.2 colorama==0.3.9 comtypes==1.1.2 conda==4.3.30 contextlib2==0.5.5 cryptography==1.8.1 cycler==0.10.0 Cython==0.25.2 cytoolz==0.8.2 dask==0.18.2 datashape==0.5.4 decorator==4.0.11 distributed==1.16.3 docutils==0.13.1 entrypoints==0.2.2 et-xmlfile==1.0.1 fastcache==1.0.2 Flask==0.12.2 Flask-Cors==3.0.2 future==0.16.0 gevent==1.2.1 greenlet==0.4.12 h5py==2.7.0 HeapDict==1.0.0 html5lib==0.999 idna==2.5 imagesize==0.7.1 imbalanced-learn==0.3.3 ipykernel==4.6.1 ipython==5.3.0 ipython-genutils==0.2.0 ipywidgets==6.0.0 isort==4.2.5 itsdangerous==0.24 jdcal==1.3 jedi==0.10.2 Jinja2==2.9.6 jmespath==0.9.3 jsonschema==2.6.0 jupyter==1.0.0 jupyter-client==5.0.1 jupyter-console==5.1.0 jupyter-core==4.3.0 Keras==2.1.6 lazy-object-proxy==1.2.2 llvmlite==0.18.0 locket==0.2.0 lxml==3.7.3 MarkupSafe==0.23 matplotlib==2.0.2 menuinst==1.4.7 mistune==0.7.4 mpmath==0.19 msgpack-python==0.4.8 multipledispatch==0.4.9 navigator-updater==0.1.0 nbconvert==5.1.1 nbformat==4.3.0 networkx==1.11 nltk==3.2.3 nose==1.3.7 notebook==5.0.0 numba==0.33.0 numexpr==2.6.2 numpy==1.14.2 numpydoc==0.6.0 odo==0.5.0 olefile==0.44 opencv-python==3.4.1 openpyxl==2.4.7 packaging==16.8 pandas==0.23.3 pandocfilters==1.4.1 partd==0.3.8 path.py==10.3.1 pathlib2==2.2.1 patsy==0.4.1 pep8==1.7.0 pickleshare==0.7.4 Pillow==4.1.1 ply==3.10 prompt-toolkit==1.0.14 psutil==5.2.2 py==1.4.33 pycosat==0.6.2 pycparser==2.17 pycrypto==2.6.1 pycurl==7.43.0 pyflakes==1.5.0 Pygments==2.2.0 pylint==1.6.4 Pympler==0.5 pyodbc==4.0.16 pyOpenSSL==17.0.0 pyparsing==2.1.4 pytest==3.0.7 python-dateutil==2.6.0 python-docx==0.8.6 pytz==2017.2 PyWavelets==0.5.2 pywin32==220 PyYAML==3.12 pyzmq==16.0.2 QtAwesome==0.4.4 qtconsole==4.3.0 QtPy==1.2.1 requests==2.14.2 rope-py3k==0.9.4.post1 s3fs==0.1.5 s3transfer==0.1.13 scikit-image==0.13.0 scikit-learn==0.19.0 scipy==1.1.0 seaborn==0.7.1 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.10.0 snowballstemmer==1.2.1 sortedcollections==0.5.3 sortedcontainers==1.5.7 sphinx==1.5.6 spyder==3.1.4 SQLAlchemy==1.1.9 statsmodels==0.8.0 sympy==1.0 tables==3.2.2 tblib==1.3.2 testpath==0.3 toolz==0.8.2 tornado==4.5.1 tqdm==4.24.0 traitlets==4.3.2 unicodecsv==0.14.1 wcwidth==0.1.7 Werkzeug==0.12.2 widgetsnbextension==2.0.0 win-unicode-console==0.5 wrapt==1.10.10 xlrd==1.0.0 XlsxWriter==0.9.6 xlwings==0.10.4 xlwt==1.2.0 zict==0.1.2

I have tried installing on another machine with almost same specs but it gives the same error for cloud pickle.

Please find attached the complete error:

(C:\Users\Faizan Saeed\Anaconda3) C:\Users\Faizan Saeed>pip install featuretools Collecting featuretools Requirement already satisfied: pympler>=0.5 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.5) Collecting cloudpickle>=0.4.0 (from featuretools) Using cached https://files.pythonhosted.org/packages/e7/bf/60ae7ec1e8c6742d2abbb6819c39a48ee796793bcdb7e1d5e41a3e379ddd/cloudpickle-0.5.3-py2.py3-none-any.whl Requirement already satisfied: psutil in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (5.2.2) Requirement already satisfied: tqdm>=4.19.2 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (4.24.0) Requirement already satisfied: s3fs>=0.1.2 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.1.5) Requirement already satisfied: future>=0.16.0 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.16.0) Collecting distributed>=1.21.8 (from featuretools) Downloading https://files.pythonhosted.org/packages/6b/09/d7e0defd4bccbc37f0588778abe548129ffea1637e46231095dd57325097/distributed-1.22.1-py2.py3-none-any.whl (480kB) 100% |████████████████████████████████| 481kB 102kB/s Requirement already satisfied: pandas>=0.20.3 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.23.3) Requirement already satisfied: scipy>=1.0.0 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (1.1.0) Requirement already satisfied: dask>=0.17.5 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.18.2) Requirement already satisfied: toolz>=0.8.2 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (0.8.2) Requirement already satisfied: numpy>=1.13.3 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (1.14.2) Requirement already satisfied: pyyaml>=3.12 in c:\users\faizan saeed\anaconda3\lib\site-packages (from featuretools) (3.12) Requirement already satisfied: boto3 in c:\users\faizan saeed\anaconda3\lib\site-packages (from s3fs>=0.1.2->featuretools) (1.7.67) Requirement already satisfied: botocore in c:\users\faizan saeed\anaconda3\lib\site-packages (from s3fs>=0.1.2->featuretools) (1.10.67) Collecting zict>=0.1.3 (from distributed>=1.21.8->featuretools) Using cached https://files.pythonhosted.org/packages/5d/c9/eddd6c9a7ebd65fc799f9b87e56b45599a4e35d66e3da2722d7fc2a89f1f/zict-0.1.3-py2.py3-none-any.whl Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in c:\users\faizan saeed\anaconda3\lib\site-packages (from distributed>=1.21.8->featuretools) (1.5.7) Requirement already satisfied: tblib in c:\users\faizan saeed\anaconda3\lib\site-packages (from distributed>=1.21.8->featuretools) (1.3.2) Requirement already satisfied: tornado>=4.5.1 in c:\users\faizan saeed\anaconda3\lib\site-packages (from distributed>=1.21.8->featuretools) (4.5.1) Requirement already satisfied: click>=6.6 in c:\users\faizan saeed\anaconda3\lib\site-packages (from distributed>=1.21.8->featuretools) (6.7) Requirement already satisfied: six in c:\users\faizan saeed\anaconda3\lib\site-packages (from distributed>=1.21.8->featuretools) (1.10.0) Collecting msgpack (from distributed>=1.21.8->featuretools) Using cached https://files.pythonhosted.org/packages/04/81/c6363198f24ec1c56e5c48ce685cb532e175125adade0cdb181c8c5fea6e/msgpack-0.5.6-cp36-cp36m-win_amd64.whl Requirement already satisfied: python-dateutil>=2.5.0 in c:\users\faizan saeed\anaconda3\lib\site-packages (from pandas>=0.20.3->featuretools) (2.6.0) Requirement already satisfied: pytz>=2011k in c:\users\faizan saeed\anaconda3\lib\site-packages (from pandas>=0.20.3->featuretools) (2017.2) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in c:\users\faizan saeed\anaconda3\lib\site-packages (from boto3->s3fs>=0.1.2->featuretools) (0.9.3) Requirement already satisfied: s3transfer<0.2.0,>=0.1.10 in c:\users\faizan saeed\anaconda3\lib\site-packages (from boto3->s3fs>=0.1.2->featuretools) (0.1.13) Requirement already satisfied: docutils>=0.10 in c:\users\faizan saeed\anaconda3\lib\site-packages (from botocore->s3fs>=0.1.2->featuretools) (0.13.1) Requirement already satisfied: heapdict in c:\users\faizan saeed\anaconda3\lib\site-packages (from zict>=0.1.3->distributed>=1.21.8->featuretools) (1.0.0) Installing collected packages: cloudpickle, zict, msgpack, distributed, featuretools Found existing installation: cloudpickle 0.2.2 Cannot uninstall 'cloudpickle'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

[kmax12]

@FaizySaeed can you try to manually uninstall cloudpickle and then reinstall featuretools?

[FaizSaeed]

Shouldn't it be that if we have a package installed we should use that rather than first uninstalling the package and then re-installing. Manually it's quite messy to uninstall a package.

[kmax12]

@FaizySaeed yes, it's not ideal. Once we verify that approach works, we can look into how to resolve it more effectively.

[Tagar]

Apache Spark (PySpark) uses cloudpickle internally. Although it comes as part of Spark Home, so you should be able to use cloudpickle that comes with Spark by putting that in PYTHONPATH, something like

export PYTHONPATH="/opt/cloudera/parcels/SPARK2/lib/spark2/python:/opt/cloudera/parcels/SPARK2/lib/spark2/python/lib/py4j-0.10.4-src.zip"

cloudpickle is also used by dask.distributed.

[FaizSaeed]

I found another way around it by using virtual environment. It's quite handy and worked for me. Instead of handling all the libraries, I simply created a separate virtual environment and installed featuretools in it.

[kmax12]

@FaizySaeed great! closing this issue.