Nike-Inc / brickflow

Pythonic Programming Framework to orchestrate jobs in Databricks Workflow
https://engineering.nike.com/brickflow/
Apache License 2.0
187 stars 44 forks source link

[BUG] WorkflowDependencySensor should not expect the bearer token to be stored in a Databricks secret #60

Closed BrendBraeckmans closed 11 months ago

BrendBraeckmans commented 11 months ago

Describe the bug In my opinion brickflow shouldn’t make use of databricks secrets as brickflow shouldn’t make this decision. If I want to have a single key vault in my cloud environment I should be able to use this one across different technologies. Say for example I’m on AWS and want to use AWS secrets manager i.s.o databricks secrets I should be able to do so. The same holds for GCP and the Secret Manager there. This is also true for Azure Key Vault even though there you can have an Azure Key Vault backed Databricks secret scope.

To Reproduce

Expected behavior Brickflow should just accept the token as an input argument and treat it as a pydantic SecretStr

Screenshots

Cloud Information

Desktop (please complete the following information):

Additional context