Enhance _wait_for_cluster_to_start to wait for library installation
Previously, the _wait_for_cluster_to_start function only checked if the cluster state was 'running'. This update adds a check to ensure all cluster libraries have been successfully installed (or skipped or restored) before proceeding. This ensures the cluster is fully ready for operations that depend on these libraries.
Added get_cluster_libraries_status to fetch the status of cluster libraries.
Updated _wait_for_cluster_to_start to wait until all libraries are in valid statuses ('INSTALLED', 'RESTORED', 'SKIPPED').
Checklist
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.
Resolves #821
Description
Enhance
_wait_for_cluster_to_startto wait for library installationPreviously, the
_wait_for_cluster_to_startfunction only checked if the cluster state was 'running'. This update adds a check to ensure all cluster libraries have been successfully installed (or skipped or restored) before proceeding. This ensures the cluster is fully ready for operations that depend on these libraries.Added
get_cluster_libraries_statusto fetch the status of cluster libraries.Updated
_wait_for_cluster_to_startto wait until all libraries are in valid statuses ('INSTALLED', 'RESTORED', 'SKIPPED').Checklist
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section.