search

snap-stanford / mars

Discovering novel cell types across heterogenous single-cell experiments
MIT License
120 stars 36 forks source link

How to use memory across multiple GPUs with MARS? #25

Closed Winbuntu closed 3 years ago

Winbuntu commented 3 years ago

Hi, I am running MARS on a dataset containing approximately 20k cells. I was running on a 1080Ti, which has 11GB of memory. But 11GB GPU memory is not enough for MARS to process this dataset, and I got a CUDA out of memory error.

Is MARS able to use memory across multiple GPUs, so we can scale up to more than 20K cells? Or is there any other way to run MARS with large dataset?

mbrbic commented 3 years ago

I was running MARS on large datasets without memory issues, but this might depend on the GPU you use. If one of your annotated datasets is very large, you can divide data in batches (currently each task is one batch -- line 259 in mars.py).

Winbuntu commented 3 years ago

We switched to a Tesla V100 with 16GB memory and the out of memory error is gone :-) It would be of great help if you could implement this data generator in MARS. It certainly helpful for large dataset.