search

uber / fiber

Distributed Computing for AI Made Simple
https://uber.github.io/fiber/
Apache License 2.0
1.04k stars 110 forks source link

Is there any plan to extend fiber beyond aws/gcp Kubernetes clusters? #38

Closed zw0610 closed 4 years ago

zw0610 commented 4 years ago

So far it seems fiber only supports aws and gcp clusters, which might not be available to users for various reasons.

I believe the kubernetes_backend.py is generic enough to launch fiber 'Process' on customized Kubernetes clusters. However, it seems the fiber cli.py does not include options other than aws and gcp. I suggest we can extend fiber cli to generic Kubernetes clusters.

calio commented 4 years ago

Hi @zw0610, you are right. Fiber CLI cli.py doesn't support launching things on generic Kubernetes clusters yet. But it's not hard to add that support. For custom k8s cluster, you can probably use DockerImageBuilder to build locally and push to your own docker registry specified by --docker-registry argument. Would you be interested to make some changes to cli.py to add this support? It would be nice if you can make the change and test it on your cluster because I don't have a custom Kubernetes cluster at hand.

zw0610 commented 4 years ago

Hi @zw0610, you are right. Fiber CLI cli.py doesn't support launching things on generic Kubernetes clusters yet. But it's not hard to add that support. For custom k8s cluster, you can probably use DockerImageBuilder to build locally and push to your own docker registry specified by --docker-registry argument. Would you be interested to make some changes to cli.py to add this support? It would be nice if you can make the change and test it on your cluster because I don't have a custom Kubernetes cluster at hand.

Sure. I'll file a pull request later.

calio commented 4 years ago

Thank you!

calio commented 4 years ago

Closing this now since the PR has been merged. It will be included in the next release soon.