astronomer / astro-provider-databricks

Orchestrate your Databricks notebooks in Airflow and execute them as Databricks Workflows
Apache License 2.0
20 stars 10 forks source link

Fix databricks-cli 0.17.6 urllib3 Retry.DEFAULT_METHOD_WHITELIST issue #40

Closed tatiana closed 1 year ago

tatiana commented 1 year ago

Since urllib 2.0 was released, our main branch has been broken in the CI:

  File /Users/tati/Code/astro-provider-databricks/src/astro_databricks/operators/notebook.py, line 236, in _get_api_client
    return ApiClient(
  File /Users/tati/Code/astro-provider-databricks/.nox/test-3-10-airflow-2-5/lib/python3.10/site-packages/databricks_cli/sdk/api_client.py, line 106, in __init__
    method_whitelist=set({'POST'}) | set(Retry.DEFAULT_METHOD_WHITELIST),
AttributeError: type object 'Retry' has no attribute 'DEFAULT_METHOD_WHITELIST'

Until there is a new databricks-cli release with this fix: https://github.com/databricks/databricks-cli/issues/634

We should pin the version of urllib, to avoid this error.

Work pending At the moment, our tests are still failing, but due to a different reason. It needs further investigation. Below are some details:

> /Users/tati/Code/astro-provider-databricks/src/astro_databricks/operators/notebook.py(209)_handle_final_state()
-> raise AirflowException(
(Pdb) l
204                 raise AirflowException(
205                     f"Databricks job failed with state {final_state}. Message: {final_state['state_message']}"
206                 )
207             if final_state["result_state"] != "SUCCESS":
208  ->             raise AirflowException(
209                     "Task failed. Final State %s. Reason: %s",
210                     final_state["result_state"],
211                     final_state["state_message"],
212                 )
213     
(Pdb) final_state
{'life_cycle_state': 'TERMINATED', 'result_state': 'FAILED', 'state_message': '', 'user_cancelled_or_timedout': False}

When checking the Databricks UI, we can see the workflow example_databricks_workflow__test_workflow_tati_1234__notebook_2 failed with error:

com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named datetime is defined