leiden alg with igraph flavor causes out of bounds freezing

RubenVanEsch commented 5 months ago

Please make sure these conditions are met

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of scanpy.
[X] (optional) I have confirmed this bug exists on the main branch of scanpy.

What happened?

was running the standard pipeline on some data and when i run sc.tl.leiden(em_adata,flavor='igraph',n_iterations=2,random_state=1653,directed=False) it spits out infinite lines of ignored exceptions. it does not actually crash the kernel, but does bog it down and causes everything to to take much more time than necesarry. I am working in a conda env on a Win 10 , 64bit, x64 system the problem also occurs using the pbmc3k dataset

Minimal code sample

# example with own data, but same happens with pbmc3k data
sc.pp.filter_cells(em_adata, min_genes=200)
sc.pp.filter_genes(em_adata, min_cells=3)
em_adata.shape
# [out] -> (42753, 21636)

sc.pp.calculate_qc_metrics(em_adata, qc_vars=["mt"], percent_top=None, log1p=False, inplace=True)
em_adata.obs["outlier_mt"] = em_adata.obs.pct_counts_mt > 15
em_adata.obs["outlier_total"] = em_adata.obs.total_counts > 30000
em_adata.obs["outlier_ngenes"] = em_adata.obs.n_genes_by_counts > 6000
em_adata = em_adata[~em_adata.obs["outlier_mt"], :]
em_adata = em_adata[~em_adata.obs["outlier_total"], :]
em_adata = em_adata[~em_adata.obs["outlier_ngenes"], :]
sc.pp.filter_genes(em_adata,min_cells=1)

sc.pp.scrublet(em_adata)
em_adata.layers['counts'] = em_adata.X.copy()
sc.pp.normalize_total(em_adata)
sc.pp.log1p(em_adata)
sc.pp.highly_variable_genes(em_adata,flavor='seurat')
sc.pl.highly_variable_genes(em_adata)
em_adata = em_adata[:, em_adata.var["highly_variable"]]
em_adata.shape
# [out] -> (41749, 1425)
sc.pp.pca(em_adata, n_comps=50)
sc.pp.neighbors(em_adata)
sc.tl.umap(em_adata)
sc.tl.leiden(em_adata,flavor='igraph',n_iterations=2,random_state=1653,directed=False)

Error output

Exception ignored in: <class 'ValueError'>
Traceback (most recent call last):
  File "numpy\\random\\mtrand.pyx", line 780, in numpy.random.mtrand.RandomState.randint
  File "numpy\\random\\_bounded_integers.pyx", line 2881, in numpy.random._bounded_integers._rand_int32
ValueError: high is out of bounds for int32

Versions

``` conda env: # Name Version Build Channel _r-mutex 1.0.0 anacondar_1 anndata 0.10.6 pypi_0 pypi anyio 4.3.0 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 py311h2bbff1b_0 array-api-compat 1.5.1 pypi_0 pypi arrow 1.3.0 pypi_0 pypi asttokens 2.4.1 pypi_0 pypi async-lru 2.0.4 py311haa95532_0 attrs 23.2.0 pypi_0 pypi babel 2.14.0 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi bleach 6.1.0 pypi_0 pypi brotli-python 1.0.9 py311hd77b12b_7 bzip2 1.0.8 h2bbff1b_5 ca-certificates 2023.12.12 haa95532_0 certifi 2024.2.2 py311haa95532_0 cffi 1.16.0 py311h2bbff1b_0 charset-normalizer 3.3.2 pypi_0 pypi colorama 0.4.6 py311haa95532_0 comm 0.2.2 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi cycler 0.12.1 pypi_0 pypi debugpy 1.8.1 pypi_0 pypi decorator 5.1.1 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 executing 2.0.1 pypi_0 pypi fastjsonschema 2.19.1 pypi_0 pypi fonttools 4.50.0 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi h11 0.14.0 pypi_0 pypi h5py 3.10.0 pypi_0 pypi httpcore 1.0.4 pypi_0 pypi httpx 0.27.0 pypi_0 pypi idna 3.6 pypi_0 pypi igraph 0.11.4 pypi_0 pypi imageio 2.34.0 pypi_0 pypi ipykernel 6.29.3 pypi_0 pypi ipython 8.22.2 pypi_0 pypi ipywidgets 8.1.2 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jedi 0.19.1 pypi_0 pypi jinja2 3.1.3 py311haa95532_0 joblib 1.3.2 pypi_0 pypi json5 0.9.22 pypi_0 pypi jsonpointer 2.4 pypi_0 pypi jsonschema 4.21.1 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi jupyter-client 8.6.1 pypi_0 pypi jupyter-core 5.7.2 pypi_0 pypi jupyter-events 0.9.1 pypi_0 pypi jupyter-lsp 2.2.4 pypi_0 pypi jupyter-server 2.13.0 pypi_0 pypi jupyter-server-terminals 0.5.3 pypi_0 pypi jupyter_client 8.6.0 py311haa95532_0 jupyter_core 5.5.0 py311haa95532_0 jupyter_events 0.8.0 py311haa95532_0 jupyter_server 2.10.0 py311haa95532_0 jupyter_server_terminals 0.4.4 py311haa95532_1 jupyterlab 4.1.5 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.25.4 pypi_0 pypi jupyterlab-widgets 3.0.10 pypi_0 pypi jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.25.1 py311haa95532_0 kiwisolver 1.4.5 pypi_0 pypi lazy-loader 0.3 pypi_0 pypi legacy-api-wrap 1.4 pypi_0 pypi leidenalg 0.10.2 pypi_0 pypi libffi 3.4.4 hd77b12b_0 libsodium 1.0.18 h62dcd97_0 llvmlite 0.42.0 pypi_0 pypi m2w64-bwidget 1.9.10 2 m2w64-bzip2 1.0.6 6 m2w64-expat 2.1.1 2 m2w64-fftw 3.3.4 6 m2w64-flac 1.3.1 # Name Version Build Channel _r-mutex 1.0.0 anacondar_1 anndata 0.10.6 pypi_0 pypi anyio 4.3.0 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 py311h2bbff1b_0 array-api-compat 1.5.1 pypi_0 pypi arrow 1.3.0 pypi_0 pypi asttokens 2.4.1 pypi_0 pypi async-lru 2.0.4 py311haa95532_0 attrs 23.2.0 pypi_0 pypi babel 2.14.0 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi bleach 6.1.0 pypi_0 pypi brotli-python 1.0.9 py311hd77b12b_7 bzip2 1.0.8 h2bbff1b_5 ca-certificates 2023.12.12 haa95532_0 certifi 2024.2.2 py311haa95532_0 cffi 1.16.0 py311h2bbff1b_0 chardet 5.2.0 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi colorama 0.4.6 py311haa95532_0 comm 0.2.2 pypi_0 pypi contourpy 1.2.0 pypi_0 pypi cycler 0.12.1 pypi_0 pypi debugpy 1.8.1 pypi_0 pypi decorator 5.1.1 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 executing 2.0.1 pypi_0 pypi fastjsonschema 2.19.1 pypi_0 pypi fonttools 4.50.0 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi h11 0.14.0 pypi_0 pypi h5py 3.10.0 pypi_0 pypi httpcore 1.0.4 pypi_0 pypi httpx 0.27.0 pypi_0 pypi idna 3.6 pypi_0 pypi igraph 0.11.4 pypi_0 pypi imageio 2.34.0 pypi_0 pypi ipykernel 6.29.3 pypi_0 pypi ipython 8.22.2 pypi_0 pypi ipywidgets 8.1.2 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jedi 0.19.1 pypi_0 pypi jinja2 3.1.3 py311haa95532_0 joblib 1.3.2 pypi_0 pypi json5 0.9.22 pypi_0 pypi jsonpointer 2.4 pypi_0 pypi jsonschema 4.21.1 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi jupyter-client 8.6.1 pypi_0 pypi jupyter-core 5.7.2 pypi_0 pypi jupyter-events 0.9.1 pypi_0 pypi jupyter-lsp 2.2.4 pypi_0 pypi jupyter-server 2.13.0 pypi_0 pypi jupyter-server-terminals 0.5.3 pypi_0 pypi jupyter_client 8.6.0 py311haa95532_0 jupyter_core 5.5.0 py311haa95532_0 jupyter_events 0.8.0 py311haa95532_0 jupyter_server 2.10.0 py311haa95532_0 jupyter_server_terminals 0.4.4 py311haa95532_1 jupyterlab 4.1.5 pypi_0 pypi jupyterlab-pygments 0.3.0 pypi_0 pypi jupyterlab-server 2.25.4 pypi_0 pypi jupyterlab-widgets 3.0.10 pypi_0 pypi jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.25.1 py311haa95532_0 kiwisolver 1.4.5 pypi_0 pypi lazy-loader 0.3 pypi_0 pypi legacy-api-wrap 1.4 pypi_0 pypi leidenalg 0.10.2 pypi_0 pypi libffi 3.4.4 hd77b12b_0 libsodium 1.0.18 h62dcd97_0 llvmlite 0.42.0 pypi_0 pypi m2w64-bwidget 1.9.10 2 m2w64-bzip2 1.0.6 6 m2w64-expat 2.1.1 2 m2w64-fftw 3.3.4 6 m2w64-flac 1.3.1 3 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gettext 0.19.7 2 m2w64-gmp 6.1.0 2 m2w64-gsl 2.1 2 m2w64-libiconv 1.14 6 m2w64-libjpeg-turbo 1.4.2 3 m2w64-libogg 1.3.2 3 m2w64-libpng 1.6.21 2 m2w64-libsndfile 1.0.26 2 m2w64-libsodium 1.0.10 2 m2w64-libtiff 4.0.6 2 m2w64-libvorbis 1.3.5 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-libxml2 2.9.3 4 m2w64-mpfr 3.1.4 4 m2w64-openblas 0.2.19 1 m2w64-pcre 8.38 2 m2w64-speex 1.2rc2 3 m2w64-speexdsp 1.2rc3 3 m2w64-tcl 8.6.5 3 m2w64-tk 8.6.5 3 m2w64-tktable 2.10 5 m2w64-wineditline 2.101 5 m2w64-xz 5.2.2 2 m2w64-zeromq 4.1.4 2 m2w64-zlib 1.2.8 10 markupsafe 2.1.5 pypi_0 pypi matplotlib 3.8.3 pypi_0 pypi matplotlib-inline 0.1.6 py311haa95532_0 mistune 3.0.2 pypi_0 pypi msys2-conda-epoch 20160418 1 natsort 8.4.0 pypi_0 pypi nbclient 0.10.0 pypi_0 pypi nbconvert 7.16.2 pypi_0 pypi nbformat 5.10.3 pypi_0 pypi nest-asyncio 1.6.0 py311haa95532_0 networkx 3.2.1 pypi_0 pypi notebook 7.1.2 pypi_0 pypi notebook-shim 0.2.4 pypi_0 pypi numba 0.59.1 pypi_0 pypi numpy 1.26.4 pypi_0 pypi openssl 3.0.13 h2bbff1b_0 overrides 7.7.0 pypi_0 pypi packaging 24.0 pypi_0 pypi pandas 2.2.1 pypi_0 pypi pandocfilters 1.5.1 pypi_0 pypi parso 0.8.3 pyhd3eb1b0_0 patsy 0.5.6 pypi_0 pypi pillow 10.2.0 pypi_0 pypi pip 23.3.1 py311haa95532_0 platformdirs 4.2.0 pypi_0 pypi plotly 5.20.0 pypi_0 pypi prometheus-client 0.20.0 pypi_0 pypi prometheus_client 0.14.1 py311haa95532_0 prompt-toolkit 3.0.43 py311haa95532_0 prompt_toolkit 3.0.43 hd3eb1b0_0 psutil 5.9.8 pypi_0 pypi pure_eval 0.2.2 pyhd3eb1b0_0 pycparser 2.21 pyhd3eb1b0_0 pygments 2.17.2 pypi_0 pypi pynndescent 0.5.11 pypi_0 pypi pyparsing 3.1.2 pypi_0 pypi pysocks 1.7.1 py311haa95532_0 python 3.11.8 he1021f5_0 python-dateutil 2.9.0.post0 pypi_0 pypi python-fastjsonschema 2.16.2 py311haa95532_0 python-json-logger 2.0.7 py311haa95532_0 pytz 2023.3.post1 py311haa95532_0 pywin32 306 pypi_0 pypi pywinpty 2.0.13 pypi_0 pypi pyyaml 6.0.1 py311h2bbff1b_0 pyzmq 25.1.2 py311hd77b12b_0 r-askpass 1.0 r36_0 r-assertthat 0.2.1 r36h6115d3f_0 r-backports 1.1.4 r36h6115d3f_0 r-base 3.6.1 hf18239d_1 r-base64enc 0.1_3 r36h6115d3f_4 r-bh 1.69.0_1 r36h6115d3f_0 r-boot 1.3_20 r36h6115d3f_0 r-broom 0.5.2 r36h6115d3f_0 r-callr 3.2.0 r36h6115d3f_0 r-caret 6.0_83 r36h6115d3f_0 r-cellranger 1.1.0 r36h6115d3f_0 r-class 7.3_15 r36h6115d3f_0 r-cli 1.1.0 r36h6115d3f_0 r-clipr 0.6.0 r36h6115d3f_0 r-cluster 2.0.8 r36h6115d3f_0 r-codetools 0.2_16 r36h6115d3f_0 r-colorspace 1.4_1 r36h6115d3f_0 r-crayon 1.3.4 r36h6115d3f_0 r-curl 3.3 r36h6115d3f_0 r-data.table 1.12.2 r36h6115d3f_0 r-dbi 1.0.0 r36h6115d3f_0 r-dbplyr 1.4.0 r36h6115d3f_0 r-dichromat 2.0_0 r36h6115d3f_4 r-digest 0.6.18 r36h6115d3f_0 r-dplyr 0.8.0.1 r36h6115d3f_0 r-ellipsis 0.1.0 r36h6115d3f_0 r-essentials 3.6.0 r36_0 r-evaluate 0.13 r36h6115d3f_0 r-fansi 0.4.0 r36h6115d3f_0 r-forcats 0.4.0 r36h6115d3f_0 r-foreach 1.4.4 r36h6115d3f_0 r-foreign 0.8_71 r36h6115d3f_0 r-formatr 1.6 r36h6115d3f_0 r-fs 1.2.7 r36h6115d3f_0 r-generics 0.0.2 r36h6115d3f_0 r-ggplot2 3.1.1 r36h6115d3f_0 r-glmnet 2.0_16 r36h6115d3f_0 r-glue 1.3.1 r36h6115d3f_0 r-gower 0.2.0 r36h6115d3f_0 r-gtable 0.3.0 r36h6115d3f_0 r-haven 2.1.0 r36h6115d3f_0 r-hexbin 1.27.2 r36h6115d3f_0 r-highr 0.8 r36h6115d3f_0 r-hms 0.4.2 r36h6115d3f_0 r-htmltools 0.3.6 r36h6115d3f_0 r-htmlwidgets 1.3 r36h6115d3f_0 r-httpuv 1.5.1 r36h6115d3f_0 r-httr 1.4.0 r36h6115d3f_0 r-ipred 0.9_8 r36h6115d3f_0 r-irdisplay 0.7.0 r36h6115d3f_0 r-irkernel 0.8.15 r36_0 r-iterators 1.0.10 r36h6115d3f_0 r-jsonlite 1.6 r36h6115d3f_0 r-kernsmooth 2.23_15 r36h6115d3f_4 r-knitr 1.22 r36h6115d3f_0 r-labeling 0.3 r36h6115d3f_4 r-later 0.8.0 r36h6115d3f_0 r-lattice 0.20_38 r36h6115d3f_0 r-lava 1.6.5 r36h6115d3f_0 r-lazyeval 0.2.2 r36h6115d3f_0 r-lubridate 1.7.4 r36h6115d3f_0 r-magrittr 1.5 r36h6115d3f_4 r-maps 3.3.0 r36h6115d3f_0 r-markdown 0.9 r36h6115d3f_0 r-mass 7.3_51.3 r36h6115d3f_0 r-matrix 1.2_17 r36h6115d3f_0 r-mgcv 1.8_28 r36h6115d3f_0 r-mime 0.6 r36h6115d3f_0 r-modelmetrics 1.2.2 r36h6115d3f_0 r-modelr 0.1.4 r36h6115d3f_0 r-munsell 0.5.0 r36h6115d3f_0 r-nlme 3.1_139 r36h6115d3f_0 r-nnet 7.3_12 r36h6115d3f_0 r-numderiv 2016.8_1 r36h6115d3f_0 r-openssl 1.3 r36h6115d3f_0 r-pbdzmq 0.3_3 r36h6115d3f_0 r-pillar 1.3.1 r36h6115d3f_0 r-pkgconfig 2.0.2 r36h6115d3f_0 r-plogr 0.2.0 r36h6115d3f_0 r-plyr 1.8.4 r36h6115d3f_0 r-prettyunits 1.0.2 r36h6115d3f_0 r-processx 3.3.0 r36h6115d3f_0 r-prodlim 2018.04.18 r36h6115d3f_0 r-progress 1.2.0 r36h6115d3f_0 r-promises 1.0.1 r36h6115d3f_0 r-ps 1.3.0 r36h6115d3f_0 r-purrr 0.3.2 r36h6115d3f_0 r-quantmod 0.4_14 r36h6115d3f_0 r-r6 2.4.0 r36h6115d3f_0 r-randomforest 4.6_14 r36h6115d3f_0 r-rbokeh 0.6.3 r36_0 r-rcolorbrewer 1.1_2 r36h6115d3f_0 r-rcpp 1.0.1 r36h6115d3f_0 r-rcpproll 0.3.0 r36h6115d3f_0 r-readr 1.3.1 r36h6115d3f_0 r-readxl 1.3.1 r36h6115d3f_0 r-recipes 0.1.5 r36h6115d3f_0 r-recommended 3.6.0 r36_0 r-rematch 1.0.1 r36h6115d3f_0 r-repr 0.19.2 r36h6115d3f_0 r-reprex 0.2.1 r36h6115d3f_0 r-reshape2 1.4.3 r36h6115d3f_0 r-rlang 0.3.4 r36h6115d3f_0 r-rmarkdown 1.12 r36h6115d3f_0 r-rpart 4.1_15 r36h6115d3f_0 r-rstudioapi 0.10 r36h6115d3f_0 r-rvest 0.3.3 r36h6115d3f_0 r-scales 1.0.0 r36h6115d3f_0 r-selectr 0.4_1 r36h6115d3f_0 r-shiny 1.3.2 r36h6115d3f_0 r-sourcetools 0.1.7 r36h6115d3f_0 r-spatial 7.3_11 r36h6115d3f_4 r-squarem 2017.10_1 r36h6115d3f_0 r-stringi 1.4.3 r36h6115d3f_0 r-stringr 1.4.0 r36h6115d3f_0 r-survival 2.44_1.1 r36h6115d3f_0 r-sys 3.2 r36h6115d3f_0 r-tibble 2.1.1 r36h6115d3f_0 r-tidyr 0.8.3 r36h6115d3f_0 r-tidyselect 0.2.5 r36h6115d3f_0 r-tidyverse 1.2.1 r36h6115d3f_0 r-timedate 3043.102 r36h6115d3f_0 r-tinytex 0.12 r36h6115d3f_0 r-ttr 0.23_4 r36h6115d3f_0 r-utf8 1.1.4 r36h6115d3f_0 r-uuid 0.1_2 r36h6115d3f_4 r-viridislite 0.3.0 r36h6115d3f_0 r-whisker 0.3_2 r36h6115d3f_4 r-withr 2.1.2 r36h6115d3f_0 r-xfun 0.6 r36h6115d3f_0 r-xml2 1.2.0 r36h6115d3f_0 r-xtable 1.8_4 r36h6115d3f_0 r-xts 0.11_2 r36h6115d3f_0 r-yaml 2.2.0 r36h6115d3f_0 r-zoo 1.8_5 r36h6115d3f_0 referencing 0.33.0 pypi_0 pypi requests 2.31.0 py311haa95532_1 rfc3339-validator 0.1.4 py311haa95532_0 rfc3986-validator 0.1.1 py311haa95532_0 rpds-py 0.18.0 pypi_0 pypi scanpy 1.10.0 pypi_0 pypi scikit-image 0.22.0 pypi_0 pypi scikit-learn 1.4.1.post1 pypi_0 pypi scikit-misc 0.3.1 pypi_0 pypi scipy 1.12.0 pypi_0 pypi seaborn 0.13.2 pypi_0 pypi send2trash 1.8.2 py311haa95532_0 session-info 1.0.0 pypi_0 pypi setuptools 68.2.2 py311haa95532_0 six 1.16.0 pyhd3eb1b0_1 sniffio 1.3.1 pypi_0 pypi soupsieve 2.5 py311haa95532_0 sqlite 3.41.2 h2bbff1b_0 stack-data 0.6.3 pypi_0 pypi stack_data 0.2.0 pyhd3eb1b0_0 statsmodels 0.14.1 pypi_0 pypi stdlib-list 0.10.0 pypi_0 pypi tenacity 8.2.3 pypi_0 pypi terminado 0.18.1 pypi_0 pypi texttable 1.7.0 pypi_0 pypi threadpoolctl 3.4.0 pypi_0 pypi tifffile 2024.2.12 pypi_0 pypi tinycss2 1.2.1 py311haa95532_0 tk 8.6.12 h2bbff1b_0 tornado 6.4 pypi_0 pypi tqdm 4.66.2 pypi_0 pypi traitlets 5.14.2 pypi_0 pypi types-python-dateutil 2.9.0.20240315 pypi_0 pypi typing-extensions 4.9.0 py311haa95532_1 typing_extensions 4.9.0 py311haa95532_1 tzdata 2024.1 pypi_0 pypi umap-learn 0.5.5 pypi_0 pypi uri-template 1.3.0 pypi_0 pypi urllib3 2.2.1 pypi_0 pypi vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2 wcwidth 0.2.13 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.7.0 pypi_0 pypi wheel 0.41.2 py311haa95532_0 widgetsnbextension 4.0.10 pypi_0 pypi win_inet_pton 1.1.0 py311haa95532_0 winpty 0.4.3 4 xz 5.4.6 h8cc25b3_0 yaml 0.2.5 he774522_0 zeromq 4.3.5 hd77b12b_0 zlib 1.2.13 h8cc25b3_0 ```

ilan-gold commented 5 months ago

@RubenVanEsch Can you provide a more minimally reproducible example?

For example, paring down a bit the above to just:

import scanpy as sc

em_adata = sc.datasets.pbmc3k()

sc.pp.pca(em_adata, n_comps=50)
sc.pp.neighbors(em_adata)
sc.tl.umap(em_adata)
sc.tl.leiden(em_adata,flavor='igraph',n_iterations=2,random_state=1653,directed=False)

does not yield any error. ~~Could you share your system info i.e., widows or mac?~~

RubenVanEsch commented 5 months ago

@ilan-gold your minimal example causes the exact same error:

Exception ignored in: <class 'ValueError'> Traceback (most recent call last): File "numpy\random\mtrand.pyx", line 780, in numpy.random.mtrand.RandomState.randint File "numpy\random\_bounded_integers.pyx", line 2881, in numpy.random._bounded_integers._rand_int32 ValueError: high is out of bounds for int32

if you are curious, it spits the error out 14.210 times (71050 lines of error message)

EDIT: the random state does not seem to matter btw, also happens with different random states

ilan-gold commented 5 months ago

Ok @RubenVanEsch we have to assume that this is a windows problem then. I think we will try to set up a test job and hopefully this catches the problem, although will likely catch others. What happens without a random_state set?

RubenVanEsch commented 5 months ago

@ilan-gold same thing without random state I think there might be some windows error relating to numpy on linux defaulting to 64 bit integer vs windows sometimes defaulting to 32 bit (those were the first couple of google hits when i searched the error). though i dont know where the seed is generated in the source code though.

ilan-gold commented 5 months ago

@RubenVanEsch Yes, and the issue there is that we're not the ones calling randint. We may be able to hack it. I'll have a look at how the pipeline errors out on our CI to maybe see where the call is coming from.

ivirshup commented 5 months ago

If the problem is windows, it's possible it will be solved by numpy 2.0. Not sure how easy the upgrade path to numpy 2.0 will be, however.

https://numpy.org/devdocs/numpy_2_0_migration_guide.html#windows-default-integer

ilan-gold commented 5 months ago

I got the test runner to do windows and while there were other errors, this one was seemingly not present: https://dev.azure.com/scverse/scanpy/_build/results?buildId=6287&view=logs&j=4eb20215-89fc-58e4-6218-2c2fa88ddf72&t=482e4b16-75d9-5f8c-9594-aadcd098d2cb&l=3977

We have a test that is strikingly similar to the more minimal example from above: https://github.com/scverse/scanpy/blob/main/scanpy/tests/notebooks/test_pbmc3k.py minus the umap. Could you try this test (which doesn't call umap) and also try it with umap so it's exactly as our little demo and let us know what you get? We also set resolution in the test. This test seems to actually pass on our CI.

In general there will be some back and forth here until we find someone near us with a windows machine since using CI to fix this problem isn't really feasible, but at least we can narrow the scope.

ivirshup commented 5 months ago

@RubenVanEsch, are you able to run this in WSL? Also, does the number you pass for random seed matter?

ilan-gold commented 5 months ago

Also, does the number you pass for random seed matter?

From @RubenVanEsch :

EDIT: the random state does not seem to matter btw, also happens with different random states

RubenVanEsch commented 5 months ago

@ivirshup @ilan-gold just got back to this, thought i could not install wsl as I am on a somewhat company restricted laptop, but turns out i can. installing it now (and probably using that from here on out). will run the tester in a bit and let you know

ivirshup commented 5 months ago

import scanpy as sc

em_adata = sc.datasets.pbmc3k()

sc.pp.pca(em_adata, n_comps=50)
sc.pp.neighbors(em_adata)
sc.tl.umap(em_adata)
sc.tl.leiden(em_adata,flavor='igraph',n_iterations=2,random_state=1653,directed=False)

@melonora, would you mind running this on your windows machine with the latest scanpy release to see if you can reproduce it?