When executing python models in submission_method: all_purpose_cluster, the dbt-databricks connector is not recognizing when the cluster is still installing libraries. This produces python models to fail intermittently when libs are needed.
Steps To Reproduce
Use an All Purpose Cluster running 14.3LTS
Configure python models to run in all_purpose_cluster mode
Create a python model that use a library that requires installation and configure it in the cluster created in 1.
Execute the python model when the All Purpose Cluster is terminated
Expected behavior
Python models waiting until libraries configured in the cluster are installed
Screenshots and log output
Based on this function, it waits until the cluster is in a running state, but this doesn't mean that the cluster is ready to use as shown in the screenshot.
https://github.com/databricks/dbt-databricks/blob/38aefbf86371341f09f39b2ae93224ba2f21270d/dbt/adapters/databricks/python_submissions.py#L404
Describe the bug
When executing python models in submission_method: all_purpose_cluster, the dbt-databricks connector is not recognizing when the cluster is still installing libraries. This produces python models to fail intermittently when libs are needed.
Steps To Reproduce
Expected behavior
Python models waiting until libraries configured in the cluster are installed
Screenshots and log output
Based on this function, it waits until the cluster is in a
runningstate, but this doesn't mean that the cluster is ready to use as shown in the screenshot.System information
The output of
dbt --version: