scverse / rapids_singlecell

Rapids_singlecell: A GPU-accelerated tool for scRNA analysis. Offers seamless scverse compatibility for efficient single-cell data processing and analysis.
https://rapids-singlecell.readthedocs.io/
MIT License
142 stars 20 forks source link

[BUG] Import time #64

Closed ivirshup closed 1 year ago

ivirshup commented 1 year ago

Describe the bug

It can take a while to import this package. Also, the behavior of the import times is odd. I think the culprit is ultimately cugraph, but I can get total import times down (by ~50%) via rearranging the import order.

Demo

Importing rsc

In [1]: %time import rapids_singlecell as rsc
CPU times: user 17.7 s, sys: 1.57 s, total: 19.3 s
Wall time: 18.3 s

Importing cugraph then rsc

In [1]: %time import cugraph, rapids_singlecell as rsc
CPU times: user 9.16 s, sys: 1.61 s, total: 10.8 s
Wall time: 9.79 s

Expected behavior A clear and concise description of what you expected to happen.

Environment details (please complete the following information):

`cupy.show_config()` ``` OS : Linux-5.15.0-82-generic-x86_64-with-glibc2.35 Python Version : 3.10.12 CuPy Version : 12.2.0 CuPy Platform : NVIDIA CUDA NumPy Version : 1.24.4 SciPy Version : 1.11.2 Cython Build Version : 0.29.36 Cython Runtime Version : None CUDA Root : /mnt/workspace/mambaforge/envs/rapids-single-cell-anndata nvcc PATH : /mnt/workspace/mambaforge/envs/rapids-single-cell-anndata/bin/nvcc CUDA Build Version : 12000 CUDA Driver Version : 12020 CUDA Runtime Version : 12020 cuBLAS Version : (available) cuFFT Version : 11008 cuRAND Version : 10303 cuSOLVER Version : (11, 5, 1) cuSPARSE Version : (available) NVRTC Version : (12, 2) Thrust Version : 200001 CUB Build Version : 200001 Jitify Build Version : 13c33ad cuDNN Build Version : 8800 cuDNN Version : 8800 NCCL Build Version : 21803 NCCL Runtime Version : 21805 cuTENSOR Version : None cuSPARSELt Build Version : None Device 0 Name : Tesla T4 Device 0 Compute Capability : 75 Device 0 PCI Bus ID : 0000:00:06.0 ```
`conda list` ``` # packages in environment at /mnt/workspace/mambaforge/envs/rapids-single-cell-anndata: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge aiohttp 3.8.5 py310h2372a71_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge anndata 0.10.0.dev86+gcb5b443 pypi_0 pypi anyio 3.7.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h27087fc_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 23.1.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py310h5764c6d_3 conda-forge arpack 3.7.0 hdefa2d7_2 conda-forge array-api-compat 1.3 pypi_0 pypi arrow 1.2.3 pyhd8ed1ab_0 conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge aws-c-auth 0.7.3 he2921ad_3 conda-forge aws-c-cal 0.6.2 hc309b26_0 conda-forge aws-c-common 0.9.0 hd590300_0 conda-forge aws-c-compression 0.2.17 h4d4d85c_2 conda-forge aws-c-event-stream 0.3.2 h2e3709c_0 conda-forge aws-c-http 0.7.12 hc865f51_1 conda-forge aws-c-io 0.13.32 h019f825_2 conda-forge aws-c-mqtt 0.9.5 h3a0376c_1 conda-forge aws-c-s3 0.3.14 h1678ad6_3 conda-forge aws-c-sdkutils 0.1.12 h4d4d85c_1 conda-forge aws-checksums 0.1.17 h4d4d85c_1 conda-forge aws-crt-cpp 0.23.0 h40cdbb9_5 conda-forge aws-sdk-cpp 1.10.57 h6f6b8fa_21 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge bleach 6.0.0 pyhd8ed1ab_0 conda-forge blosc 1.21.4 h0f2a231_0 conda-forge bokeh 3.2.2 pyhd8ed1ab_0 conda-forge boost-cpp 1.78.0 h6582d0a_3 conda-forge branca 0.6.0 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h166bdaf_9 conda-forge brotli-bin 1.0.9 h166bdaf_9 conda-forge brotlipy 0.7.0 py310h5764c6d_1005 conda-forge brunsli 0.1 h9c3ff4c_0 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.19.1 hd590300_0 conda-forge c-blosc2 2.10.2 hb4ffafa_0 conda-forge ca-certificates 2023.7.22 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pyhd8ed1ab_0 conda-forge cairo 1.16.0 hbbf8b49_1016 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py310h255011f_3 conda-forge cfitsio 4.3.0 hbdc6101_0 conda-forge charls 2.4.2 h59595ed_0 conda-forge charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge colorcet 3.0.1 pyhd8ed1ab_0 conda-forge comm 0.1.4 pyhd8ed1ab_0 conda-forge contourpy 1.1.0 py310hd41b1e2_0 conda-forge cryptography 41.0.3 py310h75e40e8_0 conda-forge cucim 23.08.00 cuda12_py310_230809_gf3a294b_0 rapidsai cuda-cccl 12.2.128 0 nvidia cuda-cudart 12.2.128 0 nvidia cuda-cudart-dev 12.2.128 0 nvidia cuda-nvcc-dev_linux-64 12.0.76 ha770c72_0 conda-forge cuda-nvcc-impl 12.0.76 h59595ed_0 conda-forge cuda-nvcc-tools 12.0.76 h59595ed_0 conda-forge cuda-nvrtc 12.2.128 0 nvidia cuda-nvtx 12.2.128 0 nvidia cuda-profiler-api 12.2.128 0 nvidia cuda-python 12.2.0 py310h79c70a0_0 nvidia cuda-version 12.0 hffde075_2 conda-forge cudf 23.08.00 cuda12_py310_230809_g8150d38e08_0 rapidsai cudf_kafka 23.08.00 cuda12_py310_230809_g8150d38e08_0 rapidsai cudnn 8.8.0.121 h264754d_2 conda-forge cugraph 23.08.00 cuda12_py310_230809_g3079227b_0 rapidsai cuml 23.08.00 cuda12_py310_230809_gd7162cdea_0 rapidsai cuproj 23.08.01 cuda12_py310_230810_g2660aba7_0 rapidsai cupy 12.2.0 py310hfc31588_0 conda-forge cusparselt 0.4.0.7 h3a97aeb_2 conda-forge cuspatial 23.08.01 cuda12_py310_230810_g2660aba7_0 rapidsai custreamz 23.08.00 cuda12_py310_230809_g8150d38e08_0 rapidsai cuxfilter 23.08.02 cuda12_py310_230811_g9d32b65_0 rapidsai cycler 0.11.0 pyhd8ed1ab_0 conda-forge cyrus-sasl 2.1.27 h54b06d7_7 conda-forge cytoolz 0.12.2 py310h2372a71_0 conda-forge dask 2023.7.1 pyhd8ed1ab_0 conda-forge dask-core 2023.7.1 pyhd8ed1ab_0 conda-forge dask-cuda 23.08.00 py310_230809_gefbd6ca_0 rapidsai dask-cudf 23.08.00 cuda12_py310_230809_g8150d38e08_0 rapidsai datashader 0.15.2 pyhd8ed1ab_0 conda-forge datashape 0.5.4 py_1 conda-forge dav1d 1.2.1 hd590300_0 conda-forge debugpy 1.6.8 py310hc6cd4ac_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge decoupler 1.4.0 pypi_0 pypi defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2023.7.1 pyhd8ed1ab_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.3 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge expat 2.5.0 hcb278e6_1 conda-forge fastrlock 0.8.2 py310hc6cd4ac_0 conda-forge fiona 1.9.4 py310h111440e_0 conda-forge fmt 9.1.0 h924138e_0 conda-forge folium 0.14.0 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.42.1 py310h2372a71_0 conda-forge fqdn 1.5.1 pyhd8ed1ab_0 conda-forge freetype 2.12.1 hca18f0e_1 conda-forge freexl 1.0.6 h166bdaf_1 conda-forge frozenlist 1.4.0 py310h2372a71_0 conda-forge fsspec 2023.6.0 pyh1a96a4e_0 conda-forge gdal 3.7.1 py310h5c4b078_7 conda-forge gdk-pixbuf 2.42.10 h6b639ba_2 conda-forge geopandas 0.13.2 pyhd8ed1ab_1 conda-forge geopandas-base 0.13.2 pyha770c72_1 conda-forge geos 3.12.0 h59595ed_0 conda-forge geotiff 1.7.1 h22adcc9_11 conda-forge gettext 0.21.1 h27087fc_0 conda-forge gflags 2.2.2 he1b5a44_1004 conda-forge giflib 5.2.1 h0b41bf4_3 conda-forge glog 0.6.0 h6f12383_0 conda-forge glpk 5.0 h445213a_0 conda-forge gmock 1.14.0 ha770c72_1 conda-forge gmp 6.2.1 h58526e2_0 conda-forge gtest 1.14.0 h00ab1b0_1 conda-forge h5py 3.9.0 pypi_0 pypi hdf4 4.2.15 h501b40f_6 conda-forge hdf5 1.14.2 nompi_h4f84152_100 conda-forge holoviews 1.17.1 pyhd8ed1ab_0 conda-forge icu 72.1 hcb278e6_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge igraph 0.10.6 h97b68dd_0 conda-forge imagecodecs 2023.8.12 py310hc929067_0 conda-forge imageio 2.31.1 pyh24c5eb1_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.0.1 pyhd8ed1ab_0 conda-forge ipykernel 6.25.1 pyh71e2992_0 conda-forge ipython 8.14.0 pyh41d4057_0 conda-forge isoduration 20.11.0 pyhd8ed1ab_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.19.0 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge json-c 0.17 h7ab15ed_0 conda-forge jsonpointer 2.0 py_0 conda-forge jsonschema 4.19.0 pyhd8ed1ab_1 conda-forge jsonschema-specifications 2023.7.1 pyhd8ed1ab_0 conda-forge jsonschema-with-format-nongpl 4.19.0 pyhd8ed1ab_1 conda-forge jupyter-server-proxy 4.0.0 pyhd8ed1ab_0 conda-forge jupyter_client 8.3.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.3.1 py310hff52083_0 conda-forge jupyter_events 0.7.0 pyhd8ed1ab_2 conda-forge jupyter_server 2.7.1 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jxrlib 1.1 h7f98852_2 conda-forge kealib 1.5.1 hcd42e92_5 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py310hd41b1e2_0 conda-forge krb5 1.21.2 h659d440_0 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.15 haa2dc70_1 conda-forge ld_impl_linux-64 2.40 h41732ed_0 conda-forge leidenalg 0.10.1 py310hc6cd4ac_0 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libabseil 20230125.3 cxx17_h59595ed_0 conda-forge libaec 1.0.6 hcb278e6_1 conda-forge libarchive 3.6.2 h039dbb9_1 conda-forge libarrow 11.0.0 hb9dc469_34_cpu conda-forge libavif 0.11.1 h8182462_2 conda-forge libblas 3.9.0 17_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_9 conda-forge libbrotlidec 1.0.9 h166bdaf_9 conda-forge libbrotlienc 1.0.9 h166bdaf_9 conda-forge libcblas 3.9.0 17_linux64_openblas conda-forge libcrc32c 1.1.2 h9c3ff4c_0 conda-forge libcublas 12.2.4.5 0 nvidia libcublas-dev 12.2.4.5 0 nvidia libcucim 23.08.00 cuda12_230809_gf3a294b_0 rapidsai libcudf 23.08.00 cuda12_230809_g8150d38e08_0 rapidsai libcudf_kafka 23.08.00 cuda12_230809_g8150d38e08_0 rapidsai libcufft 11.0.8.91 0 nvidia libcufile 1.7.1.12 0 nvidia libcufile-dev 1.7.1.12 0 nvidia libcugraph 23.08.00 cuda12_230809_g3079227b_0 rapidsai libcugraph_etl 23.08.00 cuda12_230809_g3079227b_0 rapidsai libcugraphops 23.08.01 cuda12_230815_gd75307b5_0 nvidia libcuml 23.08.00 cuda12_230809_gd7162cdea_0 rapidsai libcumlprims 23.08.00 cuda12_230809_g71c0a86_0 nvidia libcurand 10.3.3.129 0 nvidia libcurand-dev 10.3.3.129 0 nvidia libcurl 8.2.1 hca28451_0 conda-forge libcusolver 11.5.1.129 0 nvidia libcusolver-dev 11.5.1.129 0 nvidia libcusparse 12.1.2.129 0 nvidia libcusparse-dev 12.1.2.129 0 nvidia libcuspatial 23.08.01 cuda12_230810_g2660aba7_0 rapidsai libdeflate 1.18 h0b41bf4_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.5.0 hcb278e6_1 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 13.1.0 he5830b7_0 conda-forge libgdal 3.7.1 h3d2b3c2_7 conda-forge libgfortran-ng 13.1.0 h69a702a_0 conda-forge libgfortran5 13.1.0 h15d22d2_0 conda-forge libglib 2.76.4 hebfc3b9_0 conda-forge libgomp 13.1.0 he5830b7_0 conda-forge libgoogle-cloud 2.12.0 h840a212_1 conda-forge libgrpc 1.56.2 h3905398_1 conda-forge libhwloc 2.9.2 nocuda_h7313eea_1008 conda-forge libiconv 1.17 h166bdaf_0 conda-forge libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge libkml 1.3.0 h37653c0_1015 conda-forge libkvikio 23.08.00 cuda12_230809_g51a9036_0 rapidsai liblapack 3.9.0 17_linux64_openblas conda-forge libleidenalg 0.11.1 h00ab1b0_0 conda-forge libllvm14 14.0.6 hcd5def8_4 conda-forge libnetcdf 4.9.2 nompi_h9db66fd_110 conda-forge libnghttp2 1.52.0 h61bc06f_0 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libntlm 1.4 h7f98852_1002 conda-forge libnuma 2.0.16 h0b41bf4_1 conda-forge libnvjpeg 12.2.1.2 0 nvidia libopenblas 0.3.23 pthreads_h80387f5_0 conda-forge libpng 1.6.39 h753d276_0 conda-forge libpq 15.4 hfc447b1_0 conda-forge libprotobuf 4.23.3 hd1fb520_0 conda-forge libraft 23.08.00 cuda12_230809_ge588d7b5_0 rapidsai libraft-headers 23.08.00 cuda12_230809_ge588d7b5_0 rapidsai libraft-headers-only 23.08.00 cuda12_230809_ge588d7b5_0 rapidsai librdkafka 1.9.2 ha5a0de0_2 conda-forge librmm 23.08.00 cuda12_230809_gf3af0e8d_0 rapidsai librttopo 1.1.0 hb58d41b_14 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libspatialindex 1.9.3 h9c3ff4c_4 conda-forge libspatialite 5.0.1 h15f6e67_28 conda-forge libsqlite 3.43.0 h2797004_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 13.1.0 hfd8a6a1_0 conda-forge libthrift 0.18.1 h8fd135c_2 conda-forge libtiff 4.5.1 h8b53f26_1 conda-forge libutf8proc 2.8.0 h166bdaf_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libuv 1.44.2 hd590300_1 conda-forge libwebp 1.3.1 hbf2b3c1_0 conda-forge libwebp-base 1.3.1 hd590300_0 conda-forge libxcb 1.15 h0b41bf4_0 conda-forge libxgboost 1.7.4 rapidsai_hbb0ba15_6 rapidsai libxml2 2.11.5 h0d562d8_0 conda-forge libzip 1.10.1 h2629f0a_0 conda-forge libzlib 1.2.13 hd590300_5 conda-forge libzopfli 1.0.3 h9c3ff4c_0 conda-forge linkify-it-py 2.0.0 pyhd8ed1ab_0 conda-forge llvmlite 0.40.1 py310h1b8f574_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge louvain 0.8.1 py310hc6cd4ac_0 conda-forge lz4 4.3.2 py310h0cfdcf0_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge lzo 2.10 h516909a_1000 conda-forge mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge markdown 3.4.4 pyhd8ed1ab_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py310h2372a71_0 conda-forge matplotlib-base 3.7.2 py310hf38f957_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mdit-py-plugins 0.4.0 pyhd8ed1ab_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge metis 5.1.1 h59595ed_0 conda-forge mistune 3.0.1 pyhd8ed1ab_0 conda-forge mpfr 4.2.0 hb012696_0 conda-forge msgpack-python 1.0.5 py310hdf3cbec_0 conda-forge multidict 6.0.4 py310h1fa729e_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge munch 4.0.0 pyhd8ed1ab_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge natsort 8.4.0 pypi_0 pypi nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.7.4 pyhd8ed1ab_0 conda-forge nbformat 5.9.2 pyhd8ed1ab_0 conda-forge nccl 2.18.5.1 h3a97aeb_0 conda-forge ncurses 6.4 hcb278e6_0 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge nodejs 20.5.1 hf52ce11_0 conda-forge nspr 4.35 h27087fc_0 conda-forge nss 3.92 h1d7d5a4_0 conda-forge numba 0.57.1 py310h0f6aa51_0 conda-forge numpy 1.24.4 py310ha4c1d20_0 conda-forge nvcomp 2.6.1 h3a97aeb_2 conda-forge nvtx 0.2.5 py310h1fa729e_0 conda-forge openjpeg 2.5.0 hfec8fc6_2 conda-forge openslide 3.4.1 ha896ae7_9 conda-forge openssl 3.1.2 hd590300_0 conda-forge orc 1.9.0 h385abfd_1 conda-forge overrides 7.4.0 pyhd8ed1ab_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 1.5.3 py310h9b08913_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge panel 1.2.1 pyhd8ed1ab_0 conda-forge param 1.13.0 pyh1a96a4e_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.4.0 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pypi_0 pypi pcre2 10.40 hc3806b6_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 10.0.0 py310h582fbeb_0 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 3.10.0 pyhd8ed1ab_0 conda-forge pooch 1.7.0 pyha770c72_3 conda-forge poppler 23.08.0 hd18248d_0 conda-forge poppler-data 0.4.12 hd8ed1ab_0 conda-forge postgresql 15.4 h8972f4a_0 conda-forge proj 9.2.1 ha643af7_0 conda-forge prometheus_client 0.17.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge protobuf 4.23.3 py310hb875b13_0 conda-forge psutil 5.9.5 py310h1fa729e_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py-xgboost 1.7.4 rapidsai_py310h1395376_6 rapidsai pyarrow 11.0.0 py310h7f027b5_34_cpu conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pyee 8.1.0 pyhd8ed1ab_0 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pylibcugraph 23.08.00 cuda12_py310_230809_g3079227b_0 rapidsai pylibraft 23.08.00 cuda12_py310_230809_ge588d7b5_0 rapidsai pympler 1.0.1 pypi_0 pypi pynndescent 0.5.10 pypi_0 pypi pynvml 11.4.1 pyhd8ed1ab_0 conda-forge pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyppeteer 1.0.2 pyhd8ed1ab_0 conda-forge pyproj 3.6.0 py310h24ef57a_1 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.10.12 hd12c33a_0_cpython conda-forge python-confluent-kafka 1.9.2 py310h5764c6d_2 conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.18.0 pyhd8ed1ab_0 conda-forge python-igraph 0.10.6 py310h33b8572_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python_abi 3.10 3_cp310 conda-forge pytz 2023.3 pyhd8ed1ab_0 conda-forge pyviz_comms 3.0.0 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py310h0a54255_0 conda-forge pyyaml 6.0.1 py310h2372a71_0 conda-forge pyzmq 25.1.1 py310h5bbb5d0_0 conda-forge raft-dask 23.08.00 cuda12_py310_230809_ge588d7b5_0 rapidsai rapids 23.08.00 cuda12_py310_230809_g2a5b6f0_0 rapidsai rapids-singlecell 0.7.3.dev22+g167863b pypi_0 pypi rapids-xgboost 23.08.00 cuda12_py310_230809_g2a5b6f0_0 rapidsai rdma-core 28.9 h59595ed_1 conda-forge re2 2023.03.02 h8c504da_0 conda-forge readline 8.2 h8228510_1 conda-forge referencing 0.30.2 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rmm 23.08.00 cuda12_py310_230809_gf3af0e8d_0 rapidsai rocm-smi 5.6.0 h59595ed_1 conda-forge rpds-py 0.9.2 py310hcb5633a_0 conda-forge rtree 1.0.1 py310hbdcdc62_2 conda-forge s2n 1.3.49 h06160fa_0 conda-forge scanpy 1.9.4 pypi_0 pypi scikit-image 0.21.0 py310hc6cd4ac_0 conda-forge scikit-learn 1.3.0 py310hf7d194e_0 conda-forge scikit-misc 0.3.0 pypi_0 pypi scipy 1.11.2 py310ha4c1d20_0 conda-forge seaborn 0.12.2 pypi_0 pypi send2trash 1.8.2 pyh41d4057_0 conda-forge session-info 1.0.0 pypi_0 pypi setuptools 68.1.2 pyhd8ed1ab_0 conda-forge shapely 2.0.1 py310h7dcad9a_2 conda-forge simpervisor 1.0.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h9fff704_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge spdlog 1.11.0 h9b3ece8_1 conda-forge sqlite 3.43.0 h2c6b66d_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge statsmodels 0.14.0 pypi_0 pypi stdlib-list 0.9.0 pypi_0 pypi streamz 0.6.4 pyh6c4a22f_0 conda-forge suitesparse 5.10.1 h9e50725_1 conda-forge tbb 2021.10.0 h00ab1b0_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyh41d4057_0 conda-forge texttable 1.6.7 pyhd8ed1ab_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tifffile 2023.8.25 pyhd8ed1ab_0 conda-forge tiledb 2.16.3 h84d19f0_1 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py310h2372a71_0 conda-forge tqdm 4.66.1 pyhd8ed1ab_0 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge treelite 3.2.0 py310h1be96d9_0 conda-forge treelite-runtime 3.2.0 pypi_0 pypi typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzcode 2023c h0b41bf4_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge uc-micro-py 1.0.1 pyhd8ed1ab_0 conda-forge ucx 1.14.1 h195a15c_3 conda-forge ucx-proc 1.0.0 gpu rapidsai ucx-py 0.33.00 py310_230809_gea1eb8f_0 rapidsai umap-learn 0.5.3 pypi_0 pypi unicodedata2 15.0.0 py310h5764c6d_0 conda-forge uri-template 1.3.0 pyhd8ed1ab_0 conda-forge urllib3 1.26.15 pyhd8ed1ab_0 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webcolors 1.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.6.2 pyhd8ed1ab_0 conda-forge websockets 10.4 py310h5764c6d_1 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge xarray 2023.8.0 pyhd8ed1ab_0 conda-forge xerces-c 3.2.4 h8d71039_2 conda-forge xgboost 1.7.4 rapidsai_py310h1395376_6 rapidsai xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.6 h8ee46fc_0 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xyzservices 2023.7.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h166bdaf_0 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.9.2 py310h2372a71_0 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zfp 1.0.0 h27087fc_3 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.16.2 pyhd8ed1ab_0 conda-forge zlib 1.2.13 hd590300_5 conda-forge zlib-ng 2.0.7 h0b41bf4_0 conda-forge zstd 1.5.5 hfc55251_0 conda-forge ```

This is using the branch from #60

ivirshup commented 1 year ago

In scanpy, as well as in libraries like scikit-learn, we avoid long import times by deferring the import of slow packages. So we'd put the calls to import cugraph inside the functions that require it.

Intron7 commented 1 year ago

@ivirshup I removed all cugraph imports and moved them to the functions on a usecase bases. I also cleaned up some cupyx general import. This way I got the import from 10 s to 2.5 s which is still not perfect but I think reasonable.

ivirshup commented 1 year ago

:+1:

In scanpy we also ended up with a test in our test suite that checks to make sure we don't accidentally add an import to any of the worst offenders again. You may wanna check out the other things we did around this (wow, old issue):

Intron7 commented 1 year ago

Thank you I copied the test_performance.py and added cugraph & cupyx.

ivirshup commented 1 year ago

😬 How bad is cupyx? anndata will try to import that if it's available at the moment.

https://github.com/scverse/anndata/blob/2c8759d1d3b5475632dd674b553a9849e435504e/anndata/compat/__init__.py#L107-L113

I've been putting off figuring out a better way to handle all these optional dependencies, but anndata is lucky to have had quickly importing optional deps.

Especially since we're so reliant on type based dispatch.

ivirshup commented 1 year ago

Tracking this for anndata now here:

Intron7 commented 1 year ago

cupyx is a bit like scipy so it's best to import only what you need eg. from cupyx.scipy import sparse as cpx_sparse . Also the __init__.py's don't cover all available functions. You have to manually import sparse linear algebra for example.