search

STOmics / Stereopy

A toolkit of spatial transcriptomic analysis.
MIT License
198 stars 64 forks source link

Memory usage when runnin find_marker_genes #321

Closed maximelepetit closed 3 weeks ago

maximelepetit commented 2 months ago

Hello,

I'm having issues on memory usage when I run the find_marker_genes function on my dataset. My dataset:

StereoExpData object with n_cells X n_genes = 118420 X 24009
bin_type: cell_bins
offset_x = None
offset_y = None
cells: ['cell_name', 'dnbCount', 'area', 'total_counts', 'n_genes_by_counts', 'pct_counts_mt', 'leiden']
genes: ['gene_name', 'n_cells', 'n_counts', 'mean_umi']
cells_matrix = ['pca', 'umap']
cells_pairwise = ['neighbors']
key_record: {'hvg': ['highly_variable_genes'], 'pca': ['pca'], 'neighbors': ['neighbors'], 'umap': ['umap'], 'cluster': ['leiden'], 'gene_exp_cluster': ['gene_exp_leiden']}
result: ['highly_variable_genes', 'pca', 'neighbors', 'umap', 'leiden', 'gene_exp_leiden', 'pca_variance_ratio']

The memory usage goes from 15 GiB to more than 125GiB of memory. Then the jupyter kernel dies. How can i deal with ?

My conda env : 

# packages in environment at /home/labex-cortex/miniconda3/envs/st:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
alabaster                 0.7.13                   pypi_0    pypi
anndata                   0.9.2                    pypi_0    pypi
arboreto                  0.1.6                    pypi_0    pypi
arrow                     1.3.0                    pypi_0    pypi
asciitree                 0.3.3                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
async-lru                 2.0.4                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
babel                     2.14.0                   pypi_0    pypi
backcall                  0.2.0              pyhd3eb1b0_0    anaconda
biopython                 1.83                     pypi_0    pypi
bleach                    6.1.0                    pypi_0    pypi
blosc2                    2.0.0                    pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
boltons                   24.0.0                   pypi_0    pypi
ca-certificates           2023.08.22           h06a4308_0    anaconda
cell-bin                  1.3.4.1                  pypi_0    pypi
certifi                   2024.2.2                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
colorcet                  3.1.0                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
comm                      0.2.2                    pypi_0    pypi
ctxcore                   0.2.0                    pypi_0    pypi
cusingler                 1.1.0                    pypi_0    pypi
cython                    3.0.10                   pypi_0    pypi
cytoolz                   0.12.3                   pypi_0    pypi
dask                      2023.5.0                 pypi_0    pypi
dask-image                2023.3.0                 pypi_0    pypi
datashader                0.15.2                   pypi_0    pypi
datashape                 0.5.2                    pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1              pyhd3eb1b0_0    anaconda
defusedxml                0.7.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distinctipy               1.3.4                    pypi_0    pypi
distributed               2023.5.0                 pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
fastcluster               1.2.6                    pypi_0    pypi
fasteners                 0.19                     pypi_0    pypi
fbpca                     1.0                      pypi_0    pypi
flatbuffers               24.3.25                  pypi_0    pypi
fqdn                      1.5.1                    pypi_0    pypi
frozendict                2.4.1                    pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
gefpy                     1.1.9                    pypi_0    pypi
geojson                   3.1.0                    pypi_0    pypi
geosketch                 1.2                      pypi_0    pypi
gtfparse                  1.2.1                    pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
h5py                      3.8.0                    pypi_0    pypi
harmonypy                 0.0.6                    pypi_0    pypi
holoviews                 1.17.1                   pypi_0    pypi
hotspotsc                 1.1.1                    pypi_0    pypi
httpcore                  1.0.5                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
hvplot                    0.9.2                    pypi_0    pypi
idna                      3.6                      pypi_0    pypi
igraph                    0.11.4                   pypi_0    pypi
imagecodecs               2023.3.16                pypi_0    pypi
imageio                   2.31.1                   pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        7.1.0                    pypi_0    pypi
importlib-resources       6.4.0                    pypi_0    pypi
importlib_metadata        6.0.0                hd3eb1b0_0    anaconda
inflect                   7.2.0                    pypi_0    pypi
interlap                  0.2.7                    pypi_0    pypi
ipykernel                 6.29.4                   pypi_0    pypi
ipython                   8.12.3                   pypi_0    pypi
ipython-genutils          0.2.0                    pypi_0    pypi
ipywidgets                8.1.3                    pypi_0    pypi
isoduration               20.11.0                  pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.4.0                    pypi_0    pypi
joypy                     0.2.6                    pypi_0    pypi
jsonpointer               3.0.0                    pypi_0    pypi
jsonschema                4.23.0                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-bokeh             3.0.5                    pypi_0    pypi
jupyter-client            7.4.9                    pypi_0    pypi
jupyter-core              5.7.2                    pypi_0    pypi
jupyter-events            0.10.0                   pypi_0    pypi
jupyter-lsp               2.2.5                    pypi_0    pypi
jupyter-server            2.14.2                   pypi_0    pypi
jupyter-server-terminals  0.5.3                    pypi_0    pypi
jupyterlab                4.2.4                    pypi_0    pypi
jupyterlab-server         2.27.3                   pypi_0    pypi
jupyterlab-widgets        3.0.11                   pypi_0    pypi
kdepy                     1.1.0                    pypi_0    pypi
lazy-loader               0.4                      pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
leidenalg                 0.10.2                   pypi_0    pypi
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libsodium                 1.0.18               h7b6447c_0    anaconda
libstdcxx-ng              11.2.0               h1234567_1  
llvmlite                  0.39.1                   pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
loompy                    3.0.6                    pypi_0    pypi
louvain                   0.8.2                    pypi_0    pypi
lxml                      5.2.1                    pypi_0    pypi
lz4                       4.3.3                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.7.1                    pypi_0    pypi
matplotlib-inline         0.1.6            py38h06a4308_0    anaconda
matplotlib-scalebar       0.8.1                    pypi_0    pypi
more-itertools            10.2.0                   pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
msgpack                   1.0.8                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multipledispatch          1.0.0                    pypi_0    pypi
multiprocessing-on-dill   3.5.0a4                  pypi_0    pypi
natsort                   7.1.1                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
nest-asyncio              1.5.6            py38h06a4308_0    anaconda
notebook                  7.2.1                    pypi_0    pypi
numba                     0.56.4                   pypi_0    pypi
numcodecs                 0.12.1                   pypi_0    pypi
numexpr                   2.8.6                    pypi_0    pypi
numpy                     1.23.5                   pypi_0    pypi
numpy-groupies            0.9.22                   pypi_0    pypi
omnipath                  1.0.8                    pypi_0    pypi
onnxruntime               1.15.1                   pypi_0    pypi
opencv-python             4.8.0.76                 pypi_0    pypi
openssl                   3.0.13               h7f8727e_0  
overrides                 7.7.0                    pypi_0    pypi
packaging                 24.0                     pypi_0    pypi
pandas                    1.5.3                    pypi_0    pypi
pandocfilters             1.5.1                    pypi_0    pypi
panel                     0.14.4                   pypi_0    pypi
param                     1.13.0                   pypi_0    pypi
parso                     0.8.4                    pypi_0    pypi
partd                     1.4.1                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
phenograph                1.5.7                    pypi_0    pypi
pickleshare               0.7.5           pyhd3eb1b0_1003    anaconda
pillow                    10.3.0                   pypi_0    pypi
pims                      0.6.1                    pypi_0    pypi
pip                       23.3.1           py38h06a4308_0  
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
platformdirs              3.10.0           py38h06a4308_0    anaconda
plotly                    5.20.0                   pypi_0    pypi
pot                       0.9.1                    pypi_0    pypi
prometheus-client         0.20.0                   pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
protobuf                  5.26.1                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0              pyhd3eb1b0_2    anaconda
pure_eval                 0.2.2              pyhd3eb1b0_0    anaconda
py-cpuinfo                9.0.0                    pypi_0    pypi
pyarrow                   15.0.2                   pypi_0    pypi
pycirclize                1.4.0                    pypi_0    pypi
pyct                      0.5.0                    pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyscenic                  0.12.1                   pypi_0    pypi
python                    3.8.19               h955ad1f_0  
python-dateutil           2.8.2              pyhd3eb1b0_0    anaconda
python-json-logger        2.0.7                    pypi_0    pypi
pyvips                    2.2.1                    pypi_0    pypi
pyviz-comms               3.0.2                    pypi_0    pypi
pywavelets                1.4.1                    pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     24.0.1                   pypi_0    pypi
readline                  8.2                  h5eee18b_0  
referencing               0.35.1                   pypi_0    pypi
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
retrying                  1.3.4                    pypi_0    pypi
rfc3339-validator         0.1.4                    pypi_0    pypi
rfc3986-validator         0.1.1                    pypi_0    pypi
rpds-py                   0.20.0                   pypi_0    pypi
scanpy                    1.9.6                    pypi_0    pypi
scikit-image              0.21.0                   pypi_0    pypi
scikit-learn              1.3.0                    pypi_0    pypi
scipy                     1.10.1                   pypi_0    pypi
seaborn                   0.12.2                   pypi_0    pypi
send2trash                1.8.3                    pypi_0    pypi
setuptools                68.2.2           py38h06a4308_0  
shapely                   2.0.3                    pypi_0    pypi
six                       1.16.0             pyhd3eb1b0_1    anaconda
slicerator                1.1.0                    pypi_0    pypi
slideio                   2.5.0                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
sortedcontainers          2.4.0                    pypi_0    pypi
spatialpandas             0.4.9                    pypi_0    pypi
sphinx                    7.1.2                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.4                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.1                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlalchemy                1.3.24                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
squidpy                   1.2.2                    pypi_0    pypi
stack-data                0.6.3                    pypi_0    pypi
stereopy                  1.2.0                    pypi_0    pypi
sympy                     1.12                     pypi_0    pypi
tables                    3.8.0                    pypi_0    pypi
tblib                     3.0.0                    pypi_0    pypi
tenacity                  8.2.3                    pypi_0    pypi
texttable                 1.7.0                    pypi_0    pypi
tifffile                  2023.2.3                 pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
tqdm                      4.65.0                   pypi_0    pypi
traitlets                 5.14.3                   pypi_0    pypi
typeguard                 4.2.1                    pypi_0    pypi
types-python-dateutil     2.9.0.20240821           pypi_0    pypi
typing-extensions         4.11.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
umap-learn                0.5.1                    pypi_0    pypi
uri-template              1.3.0                    pypi_0    pypi
urllib3                   2.1.0                    pypi_0    pypi
validators                0.28.0                   pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
webcolors                 24.8.0                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.8.0                    pypi_0    pypi
wheel                     0.41.2           py38h06a4308_0  
widgetsnbextension        4.0.11                   pypi_0    pypi
xarray                    0.20.1                   pypi_0    pypi
xyzservices               2024.6.0                 pypi_0    pypi
xz                        5.4.6                h5eee18b_0  
yarl                      1.9.4                    pypi_0    pypi
zarr                      2.16.1                   pypi_0    pypi
zeromq                    4.3.4                h2531618_0    anaconda
zict                      3.0.0                    pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

Bests

Maxime

tanliwei-coder commented 2 months ago

Your data is very large, it really need this memory.