Closed prete closed 4 years ago
Hi,
I have the same issue with scgen batch removal notebook - it only uses a single core. Would be great if it is possible to speed it up by using multiple cores.
Thanks a lot, Veronika
Hi
Do you mean
Hi, I'm trying to run the scgen_batch_removal.ipynb example notebook and currently only runs on a single core. In another environment it used to run using multiple cores but now I am not able to reproduce it.
Is there something you could recommend?
Current logging.print_versions:
scanpy==1.4.6 anndata==0.7.1 umap==0.3.10 numpy==1.17.0 scipy==1.3.0 pandas==0.24.2 scikit-learn==0.21.2 statsmodels==0.10.0 tensorflow==1.13.0-rc2 scgen==1.1.4
Thank you!
Hi,
Is the training slow or scgen.batch_removal
? because scgen.batch_removal()
mostly uses cpu.
Mo
Would you please run it on GPU? if it is CPU then it will be slow
Training
network.train(train_data=train, n_epochs=100)
Would you please run it on GPU? if it is CPU then it will be slow
Thank you for your answer @M0hammadL, sadly I don't have access to GPUs from this environment.
I get that GPU will be faster, but can it be run on CPU using multiprocessing (more than one core)?
I think it heavily depends on you configuration for multi-process, we have not explicitly used multiprocessing in the code. Can you get access to GPU? Maybe I can help.
how big is your dataset?
Hi @M0hammadL ,
I've been using scGen
on CPUs and it has definitely ran in multiple CPU's before. Using 500K cells with 10K HVGs on a machine with 30 CPUs this step took ~ 3 hrs, which is OK for now.
I have tried to run scGen
on a GPU server in the past, but this didn't work. However, I'm happy to give it a try again. Can you provide some advice on the set up to run scGen
on GPUs?
Hi @M0hammadL ,
I've been using
scGen
on CPUs and it has definitely ran in multiple CPU's before. Using 500K cells with 10K HVGs on a machine with 30 CPUs this step took ~ 3 hrs, which is OK for now.I have tried to run
scGen
on a GPU server in the past, but this didn't work. However, I'm happy to give it a try again. Can you provide some advice on the set up to runscGen
on GPUs?
Ok, I would say first Install scgen
and then remove the tf installed by scgen
. Next, install tensorflow-gpu and check whether scgen
uses the GPU.
Hi, I'm trying to run the scgen_batch_removal.ipynb example notebook and currently only runs on a single core. In another environment it used to run using multiple cores but now I am not able to reproduce it.
Is there something you could recommend?
Current logging.print_versions:
scanpy==1.4.6 anndata==0.7.1 umap==0.3.10 numpy==1.17.0 scipy==1.3.0 pandas==0.24.2 scikit-learn==0.21.2 statsmodels==0.10.0 tensorflow==1.13.0-rc2 scgen==1.1.4
Thank you!