search

reanahub / reana-job-controller

REANA Job Controller
http://reana-job-controller.readthedocs.io/
MIT License
2 stars 38 forks source link

cephfs: make ceph details (monitors, mounts, user, secret-name) confugurable #29

Open lukasheinrich opened 7 years ago

lukasheinrich commented 7 years ago

To make REANA deployable at other site, details of the shared data mount for jobs need to be configurable. Probably a configuration file attached as a secret + a env var specifying that file is best.

tiborsimko commented 7 years ago

Do you mean to deploy only reana-job-controller component alone, or deploy the whole REANA ecosystem at another site?

If the latter case, we should rather muse about this in the REANA umbrella repository. E.g. in the longer term we'd aim for something like pip install reana[k8s,yadage,airflow] which will install all the necessary components for the given deployment scenario the use has in mind. Sharing configuration amongst REANA components would be easier in that way, indeed via configuration files and environment variables and friends.

lukasheinrich commented 7 years ago

no for now I want to deploy the entire REANA ecosystem on my own cluster that is

1) a kubernetes cluster based on openstack magnum 2) has a CephFS share (with its own credentials, i.e. username and auth key) based on OpenStack Manila

this is probably a setup that people will have on which to deploy the REANA infrastructure. Possibly it could be even more generalized to any kind of storage:

1) just one host directory in case all pods live on the same host 2) various shared volume drivers that kubernetes suppots

Kubernetes already factors out the notion of a volume nicely. This issue was about carrying this factorization into REANA as well.

tiborsimko commented 7 years ago

Good, so this issue should really go to the REANA umbrella repository rather than here. @diegodelemos started to work on parametrisable Kubernetes resources https://github.com/reanahub/reana-resources-k8s/pull/18 but this is a work-in-progress still. The volume sharing would go naturally together....

lukasheinrich commented 7 years ago

ok so should this be merged with #18 The scope is a bit larger because beyond generating k8s manifests this also requires some code changes to the actual services.