add ability to run the text rebuilder on a dask kubernetes cluster

impresso / impresso-pycommons

Python module with bits of code (objects, functions) highly reusable within impresso.

http://impresso-pycommons.rtfd.io/

GNU Affero General Public License v3.0

3 stars 3 forks source link

Closed mromanello closed 4 years ago

mromanello commented 5 years ago

this is necessary to run the rebuild script at scale on EPFL's Kubernetes cluster
it requires
- adding @SeguinBe's dask-k8 library as a dependency
- adding a flag to the text rebuilder CLI

mromanello commented 5 years ago

[x] this in turn requires us to create our own docker image and register it on the EPFL docker registry (as suggested by @SeguinBe). It will make everything much smoother on the longer run.