Memory usage for dataset

[inofechm]

Hello thank you for developing such a fantastic tool. I have been running LIANA on our HPC on individual samples with ~10-20 cell types and ~5000 cells per sample. At the CellphoneDB step my jobs keep getting killed due to memory usage, however I am providing 200G which seems quite excessive for small datasets. I wanted clarification on whether liana_wrap() by default does any parallel processing, multi-thread steps that may be crashing my jobs? Thank you!

[dbdimitrov]

Hi @inofechm,

It will not attempt parallel-processing by default. Also, sounds very unlikely to explode in RAM because of 5,000 cells. You can easily run this in local in fact.

Perhaps, you could lazy load your data and try to run it on local - e.g. HDF5Array. (I've tested liana's by sample function and it works well with lazy-loaded single-cell experiment object)

[inofechm]

Thanks, it's probably an internal issue with our HPC, closing the issue

[angelasanzo]

Hi @dbdimitrov,

Thank you for developing such an amazing tool. I have tried Liana previously with small datasets, however, now I am working with >40k cells and more than 30 cell types. I have tried downsampling and lazy loading it, however, it takes all the memory available when it finishes running cellphonedb and they get killed. I was wondering if there have been any new developments or additional advice you can provide to address this issue.

Thank you for your help.

[dbdimitrov]

Hi @angelasanzo,

If you want to run liana on a personal laptop, i recommend using liana-py. https://liana-py.readthedocs.io/en/latest/

The speed up and RAM efficiency from R is quite a jump. I explain how to convert to AnnData here: https://github.com/saezlab/liana/issues/148

Daniel

[angelasanzo]

It worked, thank you very much!