PrefectHQ / prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
https://prefect.io
Apache License 2.0
15.95k stars 1.57k forks source link

Security: Support for Secret Env Variables in Workers #9208

Open ghost opened 1 year ago

ghost commented 1 year ago

First check

Prefect Version

2.x

Describe the current behavior

Right now there is no documented way on how to add Secret Env Variables to a Kubernetes Worker.

This would be important in order to be able to change the API Key from a normal env variable into a secret env variable.

Describe the proposed behavior

If it is already possible to do this, please document it. If not, please add a method to do this.

If it is not possible to directly implement this, a workaround would be helpful. E.g. allow passing JSON Patch Customizations would also work.

Example Use

No response

Additional context

No response

billpalombi commented 1 year ago

We will absolutely do this @xyxz-web. We don't have a timeline yet, but I'll provide updates here.

ghost commented 1 year ago

Thats great to hear :-) I'll probably be able to give prefect a deeper look once this is implemented.

billpalombi commented 1 year ago

I'm glad to hear that!

Users are currently using secret string blocks to store secret information.

ghost commented 1 year ago

Users are currently using secret string blocks to store secret information.

How would you use that to pass the API Key to the pod spawned by the Kubernetes Worker?

billpalombi commented 1 year ago

Ah, my apologies, I didn't take a close enough look at your use case - it would be difficult. Block values are primary set up to be retrieved in Python.