Notebook doesn't know the name of the secret scope

[xtellurian]

The operator is creating Databricks Secret Scopes with name instance.ObjectMeta.Name and appending "_scope"

However our notebooks don't know the value of instance.ObjectMeta.Name and hence cannot construct the secret scope name.

[xtellurian]

Proposed fix:

The controller can automatically add a parameter to the job run, which can be called "_secretScope". The underscore implies a system generated parameter.

I think it's preferrable to explicitly pass the secret scope, rather than having to append _scope to the "run name"

[xtellurian]

https://github.com/microsoft/azure-databricks-operator/blob/master/controllers/notebookjob_controller_databricks.go

[xtellurian]

expect this to change in the future, but for now you can get the secret scope created for the run using:

secret_scope_name= dbutils.widgets.get("secret_scope")

then get a secret using:

secret_val= dbutils.secrets.get(scope=secret_scope_name, key="secret key")