search

scverse / scanpy

Single-cell analysis in Python. Scales to >1M cells.
https://scanpy.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.92k stars 600 forks source link

Running sc.pp.highly_variable_genes(adata, n_top_genes=5000, flavor='seurat_v3') produces scikit-misc error; package not installable with either pip or conda #3144

Closed DNApolymerase3 closed 4 months ago

DNApolymerase3 commented 4 months ago

Please make sure these conditions are met

What happened?

Hello scanpy! First time, please let me know what to fix about my question asking! When running sc.pp.highly_variable_genes I get this error "ImportError: Please install skmisc package via `pip install --user scikit-misc " I've tried numerous times to install the package through pip and conda, pip gives me a metadata error and conda can't find the package even using the command the website gives at https://anaconda.org/conda-forge/scikit-misc.

I checked around online and there was a two year old thread on the subject (https://github.com/scverse/scanpy/issues/2073), but none of the solutions worked on my machine. I tried version control of all the packages, most importantly numpy as it was mentioned as a problem. I was wondering if this you all have any newer solutions for this issue.

Due to my inexperience im really not sure what is causing this issue and therefore what to provide you all with

I also just realized that after a clean install trying to pip3 install scikit-misc returns this

(scanpy_env) user@Mac ~ % pip3 install scikit-misc
Requirement already satisfied: scikit-misc in /opt/miniconda3/envs/scanpy_env/lib/python3.9/site-packages (0.3.1) Requirement already satisfied: numpy>=1.22.3 in /opt/miniconda3/envs/scanpy_env/lib/python3.9/site-packages (from scikit-misc) (1.26.4)

Still getting the error though...

Hardware: M2 max mac, macos 15 beta (could this be it somehow?)

Minimal code sample

Original error upon running highly variable genes

<details>

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
File /opt/miniconda3/envs/scanpyenvt/lib/python3.12/site-packages/scanpy/preprocessing/_highly_variable_genes.py:66, in _highly_variable_genes_seurat_v3(adata, flavor, layer, n_top_genes, batch_key, check_values, span, subset, inplace)
     65 try:
---> 66     from skmisc.loess import loess
     67 except ImportError:

ModuleNotFoundError: No module named 'skmisc'

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
Cell In[14], line 1
----> 1 doublet_training_data = sc.pp.highly_variable_genes(adata, n_top_genes=6000, subset=True, flavor='seurat_v3')
      2 doublet_training_data

File /opt/miniconda3/envs/scanpyenvt/lib/python3.12/site-packages/legacy_api_wrap/__init__.py:80, in legacy_api.<locals>.wrapper.<locals>.fn_compatible(*args_all, **kw)
     77 @wraps(fn)
     78 def fn_compatible(*args_all: P.args, **kw: P.kwargs) -> R:
     79     if len(args_all) <= n_positional:
---> 80         return fn(*args_all, **kw)
     82     args_pos: P.args
     83     args_pos, args_rest = args_all[:n_positional], args_all[n_positional:]

File /opt/miniconda3/envs/scanpyenvt/lib/python3.12/site-packages/scanpy/preprocessing/_highly_variable_genes.py:655, in highly_variable_genes(adata, layer, n_top_genes, min_disp, max_disp, min_mean, max_mean, span, n_bins, flavor, subset, inplace, batch_key, check_values)
    653         sig = signature(_highly_variable_genes_seurat_v3)
    654         n_top_genes = cast(int, sig.parameters["n_top_genes"].default)
--> 655     return _highly_variable_genes_seurat_v3(
    656         adata,
    657         flavor=flavor,
    658         layer=layer,
    659         n_top_genes=n_top_genes,
    660         batch_key=batch_key,
    661         check_values=check_values,
    662         span=span,
    663         subset=subset,
    664         inplace=inplace,
    665     )
    667 cutoff = _Cutoffs.validate(
    668     n_top_genes=n_top_genes,
    669     min_disp=min_disp,
   (...)
    672     max_mean=max_mean,
    673 )
    674 del min_disp, max_disp, min_mean, max_mean, n_top_genes

File /opt/miniconda3/envs/scanpyenvt/lib/python3.12/site-packages/scanpy/preprocessing/_highly_variable_genes.py:68, in _highly_variable_genes_seurat_v3(adata, flavor, layer, n_top_genes, batch_key, check_values, span, subset, inplace)
     66     from skmisc.loess import loess
     67 except ImportError:
---> 68     raise ImportError(
     69         "Please install skmisc package via `pip install --user scikit-misc"
     70     )
     71 df = pd.DataFrame(index=adata.var_names)
     72 data = _get_obs_rep(adata, layer=layer)

ImportError: Please install skmisc package via `pip install --user scikit-misc

error when attempting install w/ conda


(scanpy_env) user@Mac ~ % conda install conda-forge::scikit-misc
Channels:
 - defaults
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - conda-forge::scikit-misc

Current channels:

  - defaults
  - https://conda.anaconda.org/conda-forge

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

Im having trouble reproducing the pip3 install scikit-misc error, which i believe is due to me switching between versions of numpy, heres what I had previously copy pasted

error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [32 lines of output]
 + meson setup /private/var/folders/76/3kg7fkgs3_d2gnkhxh5npt280000gn/T/pip-install-jl3ciqs5/scikit-misc_c6914ced8cef4877a07998e75e28ca3e /private/var/folders/76/3kg7fkgs3_d2gnkhxh5npt280000gn/T/pip-install-jl3ciqs5/scikit-misc_c6914ced8cef4877a07998e75e28ca3e/.mesonpy-ykxq6c1e -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/private/var/folders/76/3kg7fkgs3_d2gnkhxh5npt280000gn/T/pip-install-jl3ciqs5/scikit-misc_c6914ced8cef4877a07998e75e28ca3e/.mesonpy-ykxq6c1e/meson-python-native-file.ini
Preparing metadata (pyproject.toml) did not run successfully.

Error output

No response

Versions

``` # Name Version Build Channel absl-py 2.1.0 pyhd8ed1ab_0 conda-forge anndata 0.10.8 pypi_0 pypi anyio 4.2.0 py39hca03da5_0 appnope 0.1.2 py39hca03da5_1001 argon2-cffi 21.3.0 pyhd3eb1b0_0 argon2-cffi-bindings 21.2.0 py39h1a28f6b_0 arpack 3.9.1 nompi_h593882a_101 conda-forge array-api-compat 1.7.1 pyhd8ed1ab_0 conda-forge asttokens 2.0.5 pyhd3eb1b0_0 async-lru 2.0.4 py39hca03da5_0 attrs 23.1.0 py39hca03da5_0 babel 2.11.0 py39hca03da5_0 backcall 0.2.0 pyhd3eb1b0_0 beautifulsoup4 4.12.3 py39hca03da5_0 blas 1.0 openblas bleach 4.1.0 pyhd3eb1b0_0 blosc2 2.0.0 pypi_0 pypi brotli 1.1.0 hb547adb_1 conda-forge brotli-bin 1.1.0 hb547adb_1 conda-forge brotli-python 1.0.9 py39h313beb8_8 bzip2 1.0.8 h80987f9_6 c-ares 1.28.1 h93a5062_0 conda-forge ca-certificates 2024.7.4 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge certifi 2024.6.2 py39hca03da5_0 cffi 1.16.0 py39he153c15_0 conda-forge charset-normalizer 2.0.4 pyhd3eb1b0_0 chex 0.1.81 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge comm 0.2.1 py39hca03da5_0 contextlib2 21.6.0 pyhd8ed1ab_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cython 3.0.10 pypi_0 pypi debugpy 1.6.7 py39h313beb8_0 decorator 5.1.1 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 dm-tree 0.1.7 py39h2666b31_0 conda-forge docrep 0.3.2 pyh44b312d_0 conda-forge et_xmlfile 1.1.0 pyhd8ed1ab_0 conda-forge etils 1.6.0 pyhd8ed1ab_0 conda-forge exceptiongroup 1.2.1 pypi_0 pypi executing 0.8.3 pyhd3eb1b0_0 flax 0.6.1 pyhd8ed1ab_1 conda-forge fonttools 4.53.0 py39hfea33bf_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge fsspec 2024.6.1 pyhff2d567_0 conda-forge future 1.0.0 pyhd8ed1ab_0 conda-forge get-annotations 0.1.2 pyhd8ed1ab_0 conda-forge glib 2.80.2 h535f939_0 conda-forge glib-tools 2.80.2 h4c882b9_0 conda-forge glpk 5.0 h6d7a090_0 conda-forge gmp 6.3.0 h7bae524_2 conda-forge grpc-cpp 1.46.4 hb15be72_9 conda-forge gst-plugins-base 1.24.4 h8a8f8c8_0 conda-forge gstreamer 1.24.4 h430e707_0 conda-forge h5py 3.11.0 nompi_py39h534c8c8_102 conda-forge hdf5 1.14.3 nompi_hec07895_105 conda-forge icu 73.2 hc8870d7_0 conda-forge idna 3.7 py39hca03da5_0 igraph 0.10.13 h762ac30_0 conda-forge importlib-metadata 7.0.1 py39hca03da5_0 importlib_metadata 7.0.1 hd3eb1b0_0 importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge ipykernel 6.28.0 py39hca03da5_0 ipython 8.15.0 py39hca03da5_0 ipywidgets 8.1.2 py39hca03da5_0 jax 0.3.15 pyhd8ed1ab_0 conda-forge jaxlib 0.3.15 cpu_py39hb5f911d_3 conda-forge jedi 0.18.1 py39hca03da5_1 jinja2 3.1.4 py39hca03da5_0 joblib 1.4.2 pyhd8ed1ab_0 conda-forge json5 0.9.6 pyhd3eb1b0_0 jsonschema 4.19.2 py39hca03da5_0 jsonschema-specifications 2023.7.1 py39hca03da5_0 jupyter 1.0.0 py39hca03da5_9 jupyter-lsp 2.2.0 py39hca03da5_0 jupyter_client 8.6.0 py39hca03da5_0 jupyter_console 6.6.3 py39hca03da5_0 jupyter_core 5.7.2 py39hca03da5_0 jupyter_events 0.10.0 py39hca03da5_0 jupyter_server 2.14.1 py39hca03da5_0 jupyter_server_terminals 0.4.4 py39hca03da5_1 jupyterlab 4.0.11 py39hca03da5_0 jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.25.1 py39hca03da5_0 jupyterlab_widgets 3.0.10 py39hca03da5_0 kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.21.3 h237132a_0 conda-forge lcms2 2.16 ha0e7c42_0 conda-forge leidenalg 0.10.2 py39hf3050f2_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20220623.0 cxx17_h28b99d4_6 conda-forge libaec 1.1.3 hebf3989_0 conda-forge libblas 3.9.0 16_osxarm64_openblas conda-forge libbrotlicommon 1.1.0 hb547adb_1 conda-forge libbrotlidec 1.1.0 hb547adb_1 conda-forge libbrotlienc 1.1.0 hb547adb_1 conda-forge libcblas 3.9.0 16_osxarm64_openblas conda-forge libclang-cpp15 15.0.7 default_he012953_5 conda-forge libclang13 18.1.7 default_hb9c8b4a_0 conda-forge libcurl 8.8.0 h7b6f9a7_1 conda-forge libcxx 17.0.6 h0812c0d_3 conda-forge libdeflate 1.20 h93a5062_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h93a5062_2 conda-forge libffi 3.4.4 hca03da5_1 libgfortran 5.0.0 13_2_0_hd922786_3 conda-forge libgfortran5 13.2.0 hf226fd6_3 conda-forge libglib 2.80.2 h535f939_0 conda-forge libiconv 1.17 h0d3ecfb_2 conda-forge libintl 0.22.5 h8fbad5d_2 conda-forge libintl-devel 0.22.5 h8fbad5d_2 conda-forge libjpeg-turbo 3.0.0 hb547adb_1 conda-forge liblapack 3.9.0 16_osxarm64_openblas conda-forge libleidenalg 0.11.1 h1995070_0 conda-forge libllvm11 11.1.0 h12f7ac0_6 libllvm15 15.0.7 h2621b3d_4 conda-forge libllvm18 18.1.7 hdac5640_0 conda-forge libnghttp2 1.58.0 ha4dd798_1 conda-forge libogg 1.3.5 h1a28f6b_1 libopenblas 0.3.21 h269037a_0 libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.43 h091b4b1_0 conda-forge libpq 16.3 h7afe498_0 conda-forge libprotobuf 3.21.12 ha614eb4_2 conda-forge libsodium 1.0.18 h1a28f6b_0 libsqlite 3.46.0 hfb93653_0 conda-forge libssh2 1.11.0 h7a5bd25_0 conda-forge libtiff 4.6.0 h07db509_3 conda-forge libvorbis 1.3.7 h1a28f6b_0 libwebp-base 1.4.0 h93a5062_0 conda-forge libxcb 1.15 hf346824_0 conda-forge libxml2 2.12.7 ha661575_1 conda-forge libzlib 1.2.13 hfb2fe0b_6 conda-forge lightning-utilities 0.11.3.post0 pyhd8ed1ab_0 conda-forge llvm-openmp 14.0.6 hc6e5704_0 llvmlite 0.43.0 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py39h80987f9_0 matplotlib-base 3.5.3 py39ha500c34_2 conda-forge matplotlib-inline 0.1.6 py39hca03da5_0 mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mistune 2.0.4 py39hca03da5_0 ml-collections 0.1.1 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.8 py39ha1e04a5_0 conda-forge mudata 0.2.3 pyhd8ed1ab_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.3.0 hd1853d3_4 conda-forge mysql-libs 8.3.0 hf036fc4_4 conda-forge natsort 8.4.0 pyhd8ed1ab_0 conda-forge nbclient 0.8.0 py39hca03da5_0 nbconvert 7.10.0 py39hca03da5_0 nbformat 5.9.2 py39hca03da5_0 ncurses 6.4 h313beb8_0 nest-asyncio 1.6.0 py39hca03da5_0 networkx 3.2.1 pypi_0 pypi ninja 1.12.1 h420ef59_0 conda-forge notebook 7.0.8 py39hca03da5_0 notebook-shim 0.2.3 py39hca03da5_0 nspr 4.35 h313beb8_0 nss 3.100 hc6e9f88_0 conda-forge numba 0.60.0 pypi_0 pypi numpy 2.0.0 pypi_0 pypi numpyro 0.13.2 pyhd8ed1ab_0 conda-forge openjpeg 2.5.2 h9f1df11_0 conda-forge openpyxl 3.1.4 py39hfb846b4_0 conda-forge openssl 3.3.1 hfb2fe0b_1 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge optax 0.2.2 pyhd8ed1ab_0 conda-forge overrides 7.4.0 py39hca03da5_0 packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py39h998126f_1 conda-forge pandocfilters 1.5.0 pyhd3eb1b0_0 parso 0.8.3 pyhd3eb1b0_0 patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.43 h26f9a81_0 conda-forge pexpect 4.8.0 pyhd3eb1b0_3 pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 10.3.0 py39h3352c98_0 conda-forge pip 24.0 py39hca03da5_0 platformdirs 3.10.0 py39hca03da5_0 ply 3.11 py39hca03da5_0 prometheus_client 0.14.1 py39hca03da5_0 prompt-toolkit 3.0.43 py39hca03da5_0 prompt_toolkit 3.0.43 hd3eb1b0_0 psutil 5.9.0 py39h1a28f6b_0 pthread-stubs 0.4 h27ca646_1001 conda-forge ptyprocess 0.7.0 pyhd3eb1b0_2 pure_eval 0.2.2 pyhd3eb1b0_0 pycparser 2.22 pyhd8ed1ab_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pynndescent 0.5.13 pyhff2d567_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyqt 5.15.10 py39h313beb8_0 pyqt5-sip 12.13.0 py39h80987f9_0 pyro-api 0.1.2 pyhd8ed1ab_0 conda-forge pyro-ppl 1.8.6 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 py39hca03da5_0 python 3.9.18 hd7ebdb9_1_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.16.2 py39hca03da5_0 python-igraph 0.11.5 py39hd012b80_1 conda-forge python-json-logger 2.0.7 py39hca03da5_0 python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytorch 1.12.1 cpu_py39h6ba7f14_0 pytorch-lightning 1.9.4 pyhd8ed1ab_1 conda-forge pytz 2024.1 py39hca03da5_0 pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 25.1.2 py39h313beb8_0 qt-main 5.15.8 hf679f28_21 conda-forge qtconsole 5.5.1 py39hca03da5_0 qtpy 2.4.1 py39hca03da5_0 re2 2023.02.01 hb7217d7_0 conda-forge readline 8.2 h1a28f6b_0 referencing 0.30.2 py39hca03da5_0 requests 2.32.2 py39hca03da5_0 rfc3339-validator 0.1.4 py39hca03da5_0 rfc3986-validator 0.1.1 py39hca03da5_0 rich 13.7.1 pyhd8ed1ab_0 conda-forge rpds-py 0.10.6 py39hf0e4da2_0 scanpy 1.8.1 pypi_0 pypi scikit-learn 1.1.2 py39h598ef33_0 conda-forge scikit-misc 0.3.1 pypi_0 pypi scipy 1.13.1 py39h3d5391c_0 conda-forge scvi-tools 0.20.3 pyhd8ed1ab_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge send2trash 1.8.2 py39hca03da5_0 session-info 1.0.0 pyhd8ed1ab_0 conda-forge setuptools 69.5.1 py39hca03da5_0 sinfo 0.3.4 pypi_0 pypi sip 6.7.12 py39h313beb8_0 six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.3.0 py39hca03da5_0 soupsieve 2.5 py39hca03da5_0 sqlite 3.45.3 h80987f9_0 stack_data 0.2.0 pyhd3eb1b0_0 statsmodels 0.14.2 py39h161d348_0 conda-forge stdlib-list 0.10.0 pyhd8ed1ab_0 conda-forge tbb 2021.8.0 h48ca7d4_0 terminado 0.17.1 py39hca03da5_0 texttable 1.7.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge tinycss2 1.2.1 py39hca03da5_0 tk 8.6.14 h6ba3021_0 tomli 2.0.1 py39hca03da5_0 toolz 0.12.1 pyhd8ed1ab_0 conda-forge torchmetrics 1.0.3 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py39h80987f9_0 tqdm 4.66.4 pyhd8ed1ab_0 conda-forge traitlets 5.14.3 py39hca03da5_0 typing-extensions 4.12.2 hd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024a h04d1e81_0 umap-learn 0.5.6 pypi_0 pypi unicodedata2 15.1.0 py39h0f82c59_0 conda-forge urllib3 2.2.2 py39hca03da5_0 wcwidth 0.2.5 pyhd3eb1b0_0 webencodings 0.5.1 py39hca03da5_1 websocket-client 1.8.0 py39hca03da5_0 wheel 0.43.0 py39hca03da5_0 widgetsnbextension 4.0.10 py39hca03da5_0 xlrd 1.2.0 pyh9f0ad1d_1 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.4.6 h80987f9_1 yaml 0.2.5 h3422bc3_2 conda-forge zeromq 4.3.5 h313beb8_0 zipp 3.19.2 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hfb2fe0b_6 conda-forge zstd 1.5.6 hb46c0d2_0 conda-forge ``` Im so sorry somethine else is wrong so scanpy.logging.print_versions() doesn't work at the moment. I'll fix it and update the post
flying-sheep commented 4 months ago

I think you could get better help on Stackoverflow or https://discuss.scverse.org/

If pip reports “Requirement already satisfied” but scanpy can’t import it, then the environment you run pip on is not the same you run scanpy in (or the environment is broken and something caused the metadata to be there while the actual package isn’t)

If you run pip --version, it should tell you its location. That location will probably be different from /opt/miniconda3/envs/scanpyenvt/, so you need to make sure you use the correct pip, either by doing

/opt/miniconda3/envs/scanpyenvt/bin/pip install ...

or by activating it.

As said, if you have questions, please go to one of the forums that have a lot of people who can help you!