search

scverse / scanpy

Single-cell analysis in Python. Scales to >1M cells.
https://scanpy.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.89k stars 595 forks source link

Enable GPU accelerated CI Jobs #1793

Closed Zethson closed 6 months ago

Zethson commented 3 years ago

A lot of computing will be GPU accelerated in the near and distant future. ScanPy is no exception as was recently demonstrated by NVIDIA. We got a nice PR (https://github.com/theislab/scanpy/pull/1533) to get started, but we are hesitant to merge it since we cannot yet cover GPU accelerated algorithms with our CI.

Zethson commented 3 years ago

Resources that may help:

LouisFaure commented 3 years ago

In case this helps, all gpu accelerated code implemented in scanpy use rapids related packages, which can be easily deployed by using their docker images and are updated on a regular basis! There is the choice of multiple os and python versions, although windows is not present.

aktech commented 3 years ago

Hi @Zethson I am the creator of Cirun.io, "GPU" and "CI" caught my eye.

FWIW I'll share my two cents. I created a service for problems like these, which is basically running custom machines (including GPUs) in GitHub Actions: https://cirun.io/

It is used in multiple open source projects needing GPU support like the following:

https://github.com/pystatgen/sgkit/ https://github.com/qutip/qutip-cupy

It is fairly simple to setup, all you need is a cloud account (AWS or GCP) and a simple yaml file describing what kind of machines you need and Cirun will spin up ephemeral machines on your cloud for GitHub Actions to run. It's native to GitHub ecosystem, which mean you can see logs/trigger in the Github's interface itself, just like any Github Action run.

Also, note that Cirun is free for Open source projects. (You only pay to your cloud provider for machine usage)

Zethson commented 3 years ago

Hi @aktech,

thank you for your comment. I only see support for AWS, GCP and DO. For us it would be much easier to stick with our Azure setup (payments etc much easier). Is there a way to use it with Azure?

aktech commented 3 years ago

Thanks for your response. As of today that's correct: AWS, GCP and DO. Azure support is a work in progress at the moment. It should be available by the end of this month most likely. Do you have a hard deadline on this?

Zethson commented 3 years ago

Thanks for your response. As of today that's correct: AWS, GCP and DO. Azure support is a work in progress at the moment. It should be available by the end of this month most likely. Do you have a hard deadline on this?

No we don't. But before using it Cirun I would have to look at more closely first.

ghost commented 2 years ago

Hi @Zethson, I'm curious whether you have a status update on this. Would be really excited to have GPU-accelerated Leiden, but I understand the issues you mention here. Thanks!

Zethson commented 2 years ago

@maarten-hifibio we are indeed actively working on this again. Feel free to join our zulip https://scverse.zulipchat.com/login/ and ping me. I can add you to the conversation.

LouisFaure commented 2 years ago

@maarten-hifibio in the mean time if you need it I have just made some GPU wrappers available on the following gist: https://gist.github.com/LouisFaure/9302aa140d7989a25ed2a44b1ce741e8

It includes one for leiden that would exactly act like sc.tl.leiden (it is part of my PR referenced here)

ghost commented 2 years ago

Thank you very much, @LouisFaure. This is super helpful!

Zethson commented 6 months ago

Closing in favor of https://github.com/scverse/rapids_singlecell 🚀