search

galaxyproject / training-material

A collection of Galaxy-related training material
https://training.galaxyproject.org
MIT License
304 stars 894 forks source link

Jupyter notebook (JupyLab tool) resources limit? #4457

Open wee-snufkin opened 11 months ago

wee-snufkin commented 11 months ago

I’m doing an analysis using salmon alevin command and after running the command the process stops at around 25% of analysed cells (that’s around 6500 cells - I know that from the output log). It happens when I’m doing it in jupyter notebook (JupyLab tool) but the notebook cell has [*] next to it suggesting that it’s still running (but no new log output is displayed). But when I run the same command with the same input files locally (VSC) the analysis goes to completion. I had to massively down sample the dataset for that command to be executed successfully on .eu server. Interestingly, the same dataset still fails on .org server. I have no idea what causes that - is there a resource limit on interactive jupyter notebook? Well, I used monocle3 and did huge dimensionality reduction in jupyter notebook which seemed to be much more computationally intensive and took up to 15mins - but it worked, in contrast to alevin now...

bgruening commented 11 months ago

This really depends on the server you are using. Can you please ask those questions at help.galaxyproject.org. In general ITs are limited. We offer salmon alevin as tool that gets more resources maybe you can use that? See you at help.galaxyproject.org.

nomadscientist commented 11 months ago

It doesn't make sense that the resources are that low, that works on Commandline on an average or indeed poor laptop. She's not putting in a giant sample there, it should be easily doable. Something else is amiss

bgruening commented 11 months ago

I can check the notebook if you like and can share it. We need to restrict the ITs a bit, this ensures that more people can run ITs in parallel.

nomadscientist commented 11 months ago

This would be awesome - @wee-snufkin can you send him your notebook?

natefoo commented 11 months ago

The resources allocated for Jupyter on usegalaxy.org are very very limited (1.5 CPUs, 1.5 GB of memory, and a 12 hour max runtime). I can increase that with some idea of how much might be needed. Like @bgruening mentioned though, this has to be kept intentionally low in order to prevent abuse. I think with some work I can grant increased resources to individual users, however.

wee-snufkin commented 10 months ago

Above it's the tutorial with the notebook but for your convenience you can download it directly from here