search

OpenMined / PySyft

Perform data science on data that remains in someone else's server
https://www.openmined.org/
Apache License 2.0
9.47k stars 1.99k forks source link

Setup GitHub CI Runner on AWS #5181

Closed madhavajay closed 2 years ago

madhavajay commented 3 years ago

Description

To allow us to do special integration tests or GPU tests we need some custom runners on AWS. In addition it might be possible to speed up some of our normal tests using these runners.

Definition of Done

The first example CI runner and token is setup and running in GitHub CI. Then we can discuss special integration tests and GPU tests.

madhavajay commented 3 years ago

https://www.lotharschulz.info/2019/12/09/github-action-self-hosted-runners-on-aws-incl-spot-instances/ https://docs.github.com/en/actions/hosting-your-own-runners

madhavajay commented 3 years ago

https://github.com/actions/runner/issues/845

JTunis commented 3 years ago

I'm excited for LambCI to be released. Not a solution right now, but good to keep an eye on!

madhavajay commented 3 years ago

@JTunis that looks perfect, any idea on ETA, or if we can get into a private beta?

aktech commented 3 years ago

Hi @madhavajay I am the creator of Cirun.io, "GPU support" and "CI Runner on AWS" caught my eye.

FWIW I'll share my two cents. I created a service for problems like these, which is basically running custom machines (including GPUs) in GitHub Actions: https://cirun.io/

It is used in multiple open source projects needing GPU support like the following:

It is fairly simple to setup, all you need is a cloud account (AWS or GCP) and a simple yaml file describing what kind of machines you need and Cirun will spin up ephemeral machines on your cloud for GitHub Actions to run. It's native to GitHub ecosystem, which mean you can see logs/trigger in the Github's interface itself, just like any Github Action run.

Also, note that Cirun is free for Open source projects. (You only pay to your cloud provider for machine usage)