search

astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
20 stars 10 forks source link

Getting 401 Client Error: Unauthorized #53

Closed paulg-mscience closed 12 months ago

paulg-mscience commented 1 year ago

I am trying to run the sample dag with a valid databricks connection that I use in other dags with our custom databricks operators.

The config for our databricks conn is:

{
  "token": "dapisomevalidtoken",
  "host": "https://my-databricks.cloud.databricks.com"
}

Are you expecting a different format for databricks conn?

Here is the call stack from the DatabricksWorkflowTaskGroup:

Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/astro_databricks/operators/workflow.py", line 165, in execute job = _get_job_by_name(self.databricks_job_name, jobs_api) File "/usr/local/lib/python3.10/site-packages/astro_databricks/operators/workflow.py", line 43, in _get_job_by_name jobs = jobs_api.list_jobs().get("jobs", []) File "/usr/local/lib/python3.10/site-packages/databricks_cli/jobs/api.py", line 36, in list_jobs resp = self.client.list_jobs(job_type=job_type, expand_tasks=expand_tasks, offset=offset, File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/service.py", line 341, in list_jobs return self.client.perform_query( File "/usr/local/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py", line 174, in perform_query raise requests.exceptions.HTTPError(message, response=e.response) requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url

paulg-mscience commented 12 months ago

I am closing this issue. My Databricks connection was configured per older guidance. It is now recommended token be placed in the password field instead of the connection's extra section.