az login support for job creation

[PaulMaksud]

when using az login instead of DATABRICK_TOKEN. databricks bundle deploy is able to upload code correctly on workspace but when comes to resource creation it fails with a terraform error like this :

Uploaded bundle files at /Users/25f671ac-9475-4c44-9856-524089e85c8f/.bundle/first-proj/staging/files!

Starting resource deployment
Error: terraform apply: exit status 1

Error: cannot create job: default auth: cannot configure default credentials. Config: host=https://adb-2827397339834432.12.azuredatabricks.net/, azure_use_msi=true, azure_client_id=25f671ac-9475-4c44-9856-524089e85c8f, azure_tenant_id=6e51e1ad-c54b-4b39-b598-0ffe9ae68fef. Env: DATABRICKS_HOST, ARM_USE_MSI, ARM_CLIENT_ID, ARM_TENANT_ID

  with databricks_job.batch_inference_job,
  on bundle.tf.json line 44, in resource.databricks_job.batch_inference_job:
  44:       },

Error: cannot create job: default auth: cannot configure default credentials. Config: host=https://adb-2827397339834432.12.azuredatabricks.net/, azure_use_msi=true, azure_client_id=25f671ac-9475-4c44-9856-524089e85c8f, azure_tenant_id=6e51e1ad-c54b-4b39-b598-0ffe9ae68fef. Env: DATABRICKS_HOST, ARM_USE_MSI, ARM_CLIENT_ID, ARM_TENANT_ID

  with databricks_job.model_training_job,
  on bundle.tf.json line 122, in resource.databricks_job.model_training_job:
 122:       },

Error: cannot create job: default auth: cannot configure default credentials. Config: host=https://adb-2827397339834432.12.azuredatabricks.net/, azure_use_msi=true, azure_client_id=25f671ac-9475-4c44-9856-524089e85c8f, azure_tenant_id=6e51e1ad-c54b-4b39-b598-0ffe9ae68fef. Env: DATABRICKS_HOST, ARM_USE_MSI, ARM_CLIENT_ID, ARM_TENANT_ID

  with databricks_job.write_feature_table_job,
  on bundle.tf.json line 178, in resource.databricks_job.write_feature_table_job:
 178:       }

Error: cannot create mlflow experiment: default auth: cannot configure default credentials. Config: host=https://adb-2827397339834432.12.azuredatabricks.net/, azure_use_msi=true, azure_client_id=25f671ac-9475-4c44-9856-524089e85c8f, azure_tenant_id=6e51e1ad-c54b-4b39-b598-0ffe9ae68fef. Env: DATABRICKS_HOST, ARM_USE_MSI, ARM_CLIENT_ID, ARM_TENANT_ID

  with databricks_mlflow_experiment.experiment,
  on bundle.tf.json line 183, in resource.databricks_mlflow_experiment.experiment:
 183:       }

Error: cannot create mlflow model: default auth: cannot configure default credentials. Config: host=https://adb-2827397339834432.12.azuredatabricks.net/, azure_use_msi=true, azure_client_id=25f671ac-9475-4c44-9856-524089e85c8f, azure_tenant_id=6e51e1ad-c54b-4b39-b598-0ffe9ae68fef. Env: DATABRICKS_HOST, ARM_USE_MSI, ARM_CLIENT_ID, ARM_TENANT_ID

  with databricks_mlflow_model.model,
  on bundle.tf.json line 189, in resource.databricks_mlflow_model.model:
 189:       }```

[pietern]

Thanks for reporting the issue.

Can you elaborate on the environment you're running this command in? The logs suggest you're using MSI. Is this on a CI runner, on an Azure VM, or on a personal laptop? And were you able to make this work when using DATABRICKS_TOKEN?

[PaulMaksud]

Yes indeed, I'm using an user managed identity of an azure vmss that has the contributor right on the databricks workspace as long as been declared as a service principal on the ui admin settings in that example.

For the DATABRICK_TOKEN it's working when I use my own either on local or on the vmss, I didn't try with the managed identity one (as it requires to pass by the api to get it)

But I also have the same Terraform error when i log in my local computer when I run az login and when I unset my databricks profile (that register my token)

It's the same behavior, it's able to upload the code to the .bundle folder in the workspace then fail with the above error.

[arpitjasa-db]

Hi @PaulMaksud would you mind trying this with the latest version of the CLI?

[arpitjasa-db]

Closing this issue since there's been no activity in a while, feel free to reopen with additional details if necessary!