Bug in PIP INSTALL SCIMAP

[virlyananda]

Hi all,

I'm encountering this issue upon installing with pip scimap. Any recommendation/alternative? Thanks.

Collecting sklearn<0.1,>=0.0 (from scimap) Using cached sklearn-0.0.post12.tar.gz (2.6 kB) Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [15 lines of output] The 'sklearn' PyPI package is deprecated, use 'scikit-learn' rather than 'sklearn' for pip commands.

  Here is how to fix this error in the main use cases:
  - use 'pip install scikit-learn' rather than 'pip install sklearn'
  - replace 'sklearn' by 'scikit-learn' in your pip requirements files
    (requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
  - if the 'sklearn' package is used by one of your dependencies,
    it would be great if you take some time to track which package uses
    'sklearn' instead of 'scikit-learn' and report it to their issue tracker
  - as a last resort, set the environment variable
    SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error

  More information is available at
  https://github.com/scikit-learn/sklearn-pypi-package
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

Best, Virly

[ajitjohnson]

hi @virlyananda what platform are you using? Are you installing it in a fresh conda environment? Thank you.

[virlyananda]

Hi Ajit,

I'm trying to run them on HMS O2 conda environment if possible. Yes, it's a fresh environment. Right after I created environment, I activated it and pip install scimap as mentioned. Not sure what I missed. I tried pip installing scimap on my personal macbook and it seems to be working.

Best, Virly

[ajitjohnson]

interesting. I looked at the source code. Scimap does not use sklearn<0.1,>=0.0 (from scimap). I wonder why it says so.

[ajitjohnson]

can you try pip install scimap --upgrade within the same environment? Also when you created the conda env in O2, did you use python=3.10 or above?

[virlyananda]

Hi Ajit,

Thanks for responding. Yes, I made sure the conda env is pytho 3.10 above, and I tried the ones with --upgrade version. This is what I'm getting:

Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [23 lines of output] pyproject.toml: line 7: using '[tool.sip.metadata]' to specify the project metadata is deprecated and will be removed in SIP v7.0.0, use '[project]' instead Traceback (most recent call last): File "/home/via975/.conda/envs/scimap/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in main() File "/home/via975/.conda/envs/scimap/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/home/via975/.conda/envs/scimap/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 152, in prepare_metadata_for_build_wheel whl_basename = backend.build_wheel(metadata_directory, config_settings) File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/sipbuild/api.py", line 46, in build_wheel project = AbstractProject.bootstrap('wheel', File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/sipbuild/abstract_project.py", line 92, in bootstrap project.setup(pyproject, tool, tool_description) File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/sipbuild/project.py", line 587, in setup self.apply_user_defaults(tool) File "/tmp/pip-install-17nq3mk3/pyqt6_da67b45e49594ce1ab121517c6faad49/project.py", line 66, in apply_user_defaults super().apply_user_defaults(tool) File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/pyqtbuild/project.py", line 70, in apply_user_defaults super().apply_user_defaults(tool) File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/sipbuild/project.py", line 237, in apply_user_defaults self.builder.apply_user_defaults(tool) File "/tmp/pip-build-env-rqmnadmw/overlay/lib/python3.10/site-packages/pyqtbuild/builder.py", line 69, in apply_user_defaults raise PyProjectOptionException('qmake', sipbuild.pyproject.PyProjectOptionException [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

Best, Virly

[ajitjohnson]

thanks @virlyananda This seems to be a different error now.

[ajitjohnson]

@virlyananda I think I figured out the issue. It is with pyqt6 which is a GUI for napari to work. As you are installing in HPC, you are not likely going to use Napari anyway.

I have split the installation.

See below for modified documentation:

By default, we recommend installing scimap alongside napari for visualization purposes. Please note that napari requires a GUI toolkit like PyQt. If you encounter any errors due to your operating system, you should install scimap and napari separately, following napari's documentation.

pip install scimap[napari]

If installation fails due to pyqt6; install scimap without napari. Should you need it, proceed with a custom installation of Napari.

pip install scimap

[virlyananda]

Hi Ajit,

Thank you. I tried installing with pip install scimap (without napari), but I'm still getting this error:

Preparing metadata (setup.py) ... error error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [15 lines of output] The 'sklearn' PyPI package is deprecated, use 'scikit-learn' rather than 'sklearn' for pip commands.

  Here is how to fix this error in the main use cases:
  - use 'pip install scikit-learn' rather than 'pip install sklearn'
  - replace 'sklearn' by 'scikit-learn' in your pip requirements files
    (requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
  - if the 'sklearn' package is used by one of your dependencies,
    it would be great if you take some time to track which package uses
    'sklearn' instead of 'scikit-learn' and report it to their issue tracker
  - as a last resort, set the environment variable
    SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error

  More information is available at
  https://github.com/scikit-learn/sklearn-pypi-package
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

[raymond301]

I'm trying to install, from a fresh conda env. Python=3.11

but continue to get this error, even after successfully install each package individually.

` Failed to build gensim llvmlite numba

ERROR: Could not build wheels for gensim, llvmlite, numba, which is required to install pyproject.toml-based projects `

[ajitjohnson]

@virlyananda In your case it should most certainly work as I installed on O2 yesterday. Can you type out all the commands that you use right from logging into O2 via ssh? Thanks.

[ajitjohnson]

@raymond301 I assume in your case it seems to be something related to the OS you are using. can you manually install llvmlite or numba into the environment that you created? If it installs can you check the version that got installed? Thank you.

pip install numba llvmlite
conda list

[raymond301]

$ python -m pip list Package Version

---

aiobotocore 2.5.4 aiohttp 3.9.3 aioitertools 0.11.0 aiosignal 1.3.1 anndata 0.10.6 array_api_compat 1.6 asciitree 0.3.3 asttokens 2.4.1 attrs 23.2.0 botocore 1.31.17 certifi 2024.2.2 charset-normalizer 3.3.2 click 8.1.7 click-plugins 1.1.1 cligj 0.7.2 cloudpickle 3.0.0 colorcet 3.1.0 comm 0.2.2 contourpy 1.2.1 cycler 0.12.1 dask 2024.4.0 dask-expr 1.0.9 dask-image 2023.8.1 datashader 0.16.0 debugpy 1.8.1 decorator 5.1.1 distributed 2024.4.0 docrep 0.3.2 exceptiongroup 1.2.0 executing 2.0.1 fasteners 0.19 fiona 1.9.6 fonttools 4.50.0 frozenlist 1.4.1 fsspec 2023.6.0 gensim 4.3.2 <==== geopandas 0.14.3 h5py 3.10.0 idna 3.6 igraph 0.11.4 imageio 2.34.0 importlib_metadata 7.1.0 inflect 7.2.0 ipykernel 6.29.3 ipython 8.22.2 jedi 0.19.1 Jinja2 3.1.3 jmespath 1.0.1 joblib 1.3.2 jupyter_client 8.6.1 jupyter_core 5.7.2 kiwisolver 1.4.5 lazy_loader 0.3 legacy-api-wrap 1.4 leidenalg 0.10.2 llvmlite 0.42.0 <==== locket 1.0.0 markdown-it-py 3.0.0 MarkupSafe 2.1.5 matplotlib 3.8.3 matplotlib-inline 0.1.6 matplotlib-scalebar 0.8.1 mdurl 0.1.2 more-itertools 10.2.0 msgpack 1.0.8 multidict 6.0.5 multipledispatch 1.0.0 multiscale_spatial_image 0.11.2 natsort 8.4.0 nest_asyncio 1.6.0 networkx 3.2.1 numba 0.59.1 <===== numcodecs 0.12.1 numpy 1.26.4 ome-zarr 0.8.3 omnipath 1.0.8 packaging 24.0 pandas 2.2.1 param 2.1.0 parso 0.8.3 partd 1.4.1 patsy 0.5.6 pexpect 4.9.0 pickleshare 0.7.5 pillow 10.3.0 PIMS 0.6.1 pip 24.0 platformdirs 4.2.0 prompt-toolkit 3.0.42 psutil 5.9.8 ptyprocess 0.7.0 pure-eval 0.2.2 pyarrow 15.0.2 pyct 0.5.0 pygeos 0.14 Pygments 2.17.2 pynndescent 0.5.12 pyparsing 3.1.2 pyproj 3.6.1 python-dateutil 2.9.0 pytz 2024.1 PyYAML 6.0.1 pyzmq 25.1.2 requests 2.31.0 rich 13.7.1 s3fs 2023.6.0 scanpy 1.10.0 scikit-image 0.22.0 scikit-learn 1.4.1.post1 scipy 1.13.0 seaborn 0.13.2 session_info 1.0.0 setuptools 69.2.0 shapely 2.0.3 six 1.16.0 slicerator 1.1.0 smart-open 7.0.4 sortedcontainers 2.4.0 spatial_image 0.3.0 spatialdata 0.0.15 squidpy 1.4.1 stack-data 0.6.2 statsmodels 0.14.1 stdlib-list 0.10.0 tblib 3.0.0 texttable 1.7.0 threadpoolctl 3.4.0 tifffile 2024.2.12 toolz 0.12.1 tornado 6.4 tqdm 4.66.2 traitlets 5.14.2 typeguard 4.2.1 typing_extensions 4.10.0 tzdata 2024.1 umap-learn 0.5.5 urllib3 1.26.18 validators 0.27.0 wcwidth 0.2.13 wheel 0.43.0 wrapt 1.16.0 xarray 2023.12.0 xarray-dataclasses 1.7.0 xarray-datatree 0.0.14 xarray-schema 0.0.3 xarray-spatial 0.3.7 yarl 1.9.4 zarr 2.17.1 zict 3.0.0 zipp 3.17.0

[ajitjohnson]

@raymond301 thank you. I have never encountered this issue previously. is your pip up to date? pip --version - What is your OS and version? I can try to google a bit. Thank you.

[raymond301]

$ pip --version pip 24.0 from /home/jupyter/conda_env/py311_Scimap/lib/python3.11/site-packages/pip (python 3.11) I'm using Google Cloud Jupyter notebook...so the underlying OS is:

jupyter@vm-326ab5cc-7fd2-4ef3-afd7-60b6514ee555:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu

[ajitjohnson]

thank you @raymond301 I have never used google cloud. I just tested google collab notebook https://colab.research.google.com/ and simply pip installed scimap. It asked me to restart the session as my installation overwrote some existing packages in their instance, however, on restarting the session, it worked fine.

I assume you are accessing it via CLI?

[raymond301]

Yes. So essentially, no solution, because it's an OS/ENV issue. It just doesn't like the versions installed, and tries and failes it's own wheel.

Building wheel for numba (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [854 lines of output]
      /tmp/pip-install-e64skym5/numba_5e06ce9a11794068bbc6f4ea279a0412/setup.py:87: DeprecationWarning:

        `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
        of the deprecation of `distutils` itself. It will be removed for
        Python >= 3.12. For older Python versions it will remain present.
        It is recommended to use `setuptools < 60.0` for those Python versions.
        For more details, see:
          https://numpy.org/devdocs/reference/distutils_status_migration.html

        import numpy.distutils.misc_util as np_misc
      TBB not found
      Using OpenMP from: True
      /home/jupyter/conda_env/py311_Raymond/lib/python3.11/site-packages/setuptools/__init__.py:81: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
      !!

              ********************************************************************************
              Requirements should be satisfied by a PEP 517 installer.
              If you are using pip, you can try `pip install --use-pep517`.
              ********************************************************************************

      !!
        dist.fetch_build_eggs(dist.setup_requires)
      running bdist_wheel
      running build

I'm not sure how to proceed. Thanks for trying.

[ajitjohnson]

unfortunately, seems so. I am unable to debug any further as I don't have Google Cloud :/

[virlyananda]

@virlyananda In your case it should most certainly work as I installed on O2 yesterday. Can you type out all the commands that you use right from logging into O2 via ssh? Thanks.

Hi Ajit,

Thank you so much for your prompt response. Below are my steps:

ssh myusername@o2.hms.harvard.edu srun --pty -p interactive --mem 550M -t 0-06:00 /bin/bash module purge module load miniconda3/4.10.3 source activate scimap

Check Python version within conda env.

python --version

Python 3.11.5

pip install scimap

error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully. │ exit code: 1 ╰─> [15 lines of output] The 'sklearn' PyPI package is deprecated, use 'scikit-learn' rather than 'sklearn' for pip commands.

  Here is how to fix this error in the main use cases:
  - use 'pip install scikit-learn' rather than 'pip install sklearn'
  - replace 'sklearn' by 'scikit-learn' in your pip requirements files
    (requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
  - if the 'sklearn' package is used by one of your dependencies,
    it would be great if you take some time to track which package uses
    'sklearn' instead of 'scikit-learn' and report it to their issue tracker
  - as a last resort, set the environment variable
    SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error

  More information is available at
  https://github.com/scikit-learn/sklearn-pypi-package
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed

× Encountered error while generating package metadata. ╰─> See above for output.

note: This is an issue with the package mentioned above, not pip. hint: See above for details.

I'm not sure if I missed anything. But please let me know if I missed certain steps. Thank you!

Best, Virly

[ajitjohnson]

thank you. If you notice, scimap does not have a setup.py file. It uses a different framework for installation and so your specific installation seems to pull something from else where.

Two things I did differently.

1. I have my own conda installed as opposed to using O2's conda.
2. I activate the env using conda activate scimap - Although it should not matter, conda recommends this way.

I would recommend installing your own miniconda on O2 and if you still have issues try talking to Jeremy during the computational forum (which happens on Thursdays).

[ajitjohnson]

@virlyananda can you try once again? I explicitly added scikit-learn to dependencies now so that it does not pull from any existing cache.

[virlyananda]

@virlyananda can you try once again? I explicitly added scikit-learn to dependencies now so that it does not pull from any existing cache.

Hi Ajit,

Thank you. It's working now. I also switched my miniconda on O2 before installing scimap package.

Best, Virly