search

nrdg / cloudknot

A python library to run your existing code on AWS Batch
https://nrdg.github.io/cloudknot/
Other
71 stars 17 forks source link

Kubernetes clusters #159

Open arokem opened 5 years ago

arokem commented 5 years ago

Kubernetes is an open-source container orchestration system.

Using Kubernetes, a cluster of machines can be tasked with running a container on a specified task ("Job"). For example, running a perl command to calculate pi to 2000 digits

We could use the existing machinery to create the CLI, package it into docker, and then use a template of this kind of job (run to completion) to point to all of our inputs.

Auto-scaling can be used to determine the number of needed machines on the fly.

One of the main motivations to pursue this is that it would give us the option to use other cloud providers (Kubernetes is currently supported at least on the three big ones, in addition to good support for running clusters on local HPC clusters).

For access control (e.g., access to S3 storage), we can use RBAC, or pass keys as part of the templated job configuration.

@akeshavan said she wants to implement this.

CC: @stefanv

richford commented 5 years ago

Yes! I think this is a good direction. But I will definitely need some help as I don't have experience with Kubernetes. @akeshavan, let's meet up for this sometime when we both come up for air.

arokem commented 4 years ago

This might have solved this issue: https://caliban.readthedocs.io/en/latest/