search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
202 stars 100 forks source link

Is `Prepare profile` really necessary step in the `onpush` workflow? #46

Closed Ceridan closed 3 years ago

Ceridan commented 3 years ago

In the onpush workflow we have the following step:

- name: Prepare profile
  run: |
    echo "[dbx-dev-azure]" >> ~/.databrickscfg
    echo "host = $DBX_AZURE_HOST" >> ~/.databrickscfg
    echo "token = $DBX_AZURE_TOKEN" >> ~/.databrickscfg

As I understand correctly it is required for the dbx tool. Is it possible to add support of environment variables to dbx or to pass HOST and TOKEN as an arguments?

I think it is important by two reasons:

  1. If dbx can read environment than we can remove this step and it also means that we remove one more place when we are working with sensitive data. As I know databricks-cli works correctly with the environment.
  2. Keep sensitive data in the custom local file could lead us to security issue. Yes, it should be cleaned up after our workflow run, but looks scary anyway. GitHub recommends pass secrets through Settings or Environment file.
renardeinside commented 3 years ago

That's a valid point. I'll fix this issue, allowing to simply pass the env variable, in a compatible way to databricks-cli.

renardeinside commented 3 years ago

@Ceridan from release 1.0.4 dbx supports DATABRICKS_HOST and DATABRICKS_TOKEN from env variables. Thanks a lot for suggesting this!