pachyderm / pachyderm

Data-Centric Pipelines and Data Versioning
https://www.pachyderm.com/
Apache License 2.0
6.16k stars 567 forks source link

Shared configmap and secrets between workers and pachd (potentially breaking change) #6856

Open chainlink opened 3 years ago

chainlink commented 3 years ago

Right now, config values are passed directly to pachd via env vars (ie STORAGE_BACKEND) some of these values are also mirrored in worker_rc.go in pachyderm. Further, some values are passed through Pachd to the workers (ie. WORKER_IMAGE)

Instead, we should have global configuration configmaps/secrets, which get read into pachd and the workers dynamically via envFrom This would eliminate the need to for PACHD to have env vars which only pass data to the workers

Questions:

Considerations:

For now I think we can move storage secrets over without too much trouble, but further thought is required for the configuration data

AB#150

chainlink commented 3 years ago

Shared secrets now exists for storage layer