search

theislab / scgen

Single cell perturbation prediction
https://scgen.readthedocs.io
GNU General Public License v3.0
260 stars 52 forks source link

Only runs on single core? #21

Closed prete closed 4 years ago

prete commented 4 years ago

Hi, I'm trying to run the scgen_batch_removal.ipynb example notebook and currently only runs on a single core. In another environment it used to run using multiple cores but now I am not able to reproduce it.

Is there something you could recommend?

Current logging.print_versions: scanpy==1.4.6 anndata==0.7.1 umap==0.3.10 numpy==1.17.0 scipy==1.3.0 pandas==0.24.2 scikit-learn==0.21.2 statsmodels==0.10.0 tensorflow==1.13.0-rc2 scgen==1.1.4

Thank you!

vkedlian commented 4 years ago

Hi,

I have the same issue with scgen batch removal notebook - it only uses a single core. Would be great if it is possible to speed it up by using multiple cores.

Thanks a lot, Veronika

M0hammadL commented 4 years ago

Hi

Do you mean

Hi, I'm trying to run the scgen_batch_removal.ipynb example notebook and currently only runs on a single core. In another environment it used to run using multiple cores but now I am not able to reproduce it.

Is there something you could recommend?

Current logging.print_versions: scanpy==1.4.6 anndata==0.7.1 umap==0.3.10 numpy==1.17.0 scipy==1.3.0 pandas==0.24.2 scikit-learn==0.21.2 statsmodels==0.10.0 tensorflow==1.13.0-rc2 scgen==1.1.4

Thank you!

Hi,

Is the training slow or scgen.batch_removal? because scgen.batch_removal() mostly uses cpu.

Mo

M0hammadL commented 4 years ago

Would you please run it on GPU? if it is CPU then it will be slow

M0hammadL commented 4 years ago

Training network.train(train_data=train, n_epochs=100)

Would you please run it on GPU? if it is CPU then it will be slow

prete commented 4 years ago

Thank you for your answer @M0hammadL, sadly I don't have access to GPUs from this environment.

I get that GPU will be faster, but can it be run on CPU using multiprocessing (more than one core)?

M0hammadL commented 4 years ago

I think it heavily depends on you configuration for multi-process, we have not explicitly used multiprocessing in the code. Can you get access to GPU? Maybe I can help.

how big is your dataset?

cartal commented 4 years ago

Hi @M0hammadL ,

I've been using scGen on CPUs and it has definitely ran in multiple CPU's before. Using 500K cells with 10K HVGs on a machine with 30 CPUs this step took ~ 3 hrs, which is OK for now.

I have tried to run scGen on a GPU server in the past, but this didn't work. However, I'm happy to give it a try again. Can you provide some advice on the set up to run scGen on GPUs?

M0hammadL commented 4 years ago

Hi @M0hammadL ,

I've been using scGen on CPUs and it has definitely ran in multiple CPU's before. Using 500K cells with 10K HVGs on a machine with 30 CPUs this step took ~ 3 hrs, which is OK for now.

I have tried to run scGen on a GPU server in the past, but this didn't work. However, I'm happy to give it a try again. Can you provide some advice on the set up to run scGen on GPUs?

Ok, I would say first Install scgen and then remove the tf installed by scgen. Next, install tensorflow-gpu and check whether scgen uses the GPU.