search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
202 stars 100 forks source link

Pipeline stuck at "Installing Library" because the library has been installed before on an existing cluster #70

Closed pebabion closed 3 years ago

pebabion commented 3 years ago

Environment:

I followed the cookiecutter template and successfully ran it the first time. However, on the second CI/CD run, it's stuck at the step Installing libraries.

For this, I'm using an existing cluster id instead of new cluster for each CI/CD run. When I check the cluster, I saw that the library was installed (during the 1st run) and now it's trying to install the same library again.

How should we avoid this situation?

[dbx][2021-05-18 16:49:23.342] [Run Id: 12] Current run status info - result state: None, lifecycle state: PENDING, state message: Installing libraries
[dbx][2021-05-18 16:49:28.564] [Run Id: 12] Current run status info - result state: None, lifecycle state: PENDING, state message: Installing libraries
[dbx][2021-05-18 16:49:33.787] [Run Id: 12] Current run status info - result state: None, lifecycle state: PENDING, state message: Installing libraries
[dbx][2021-05-18 16:49:39.012] [Run Id: 12] Current run status info - result state: None, lifecycle state: PENDING, state message: Installing libraries
[dbx][2021-05-18 16:49:44.276] [Run Id: 12] Current run status info - result state: None, lifecycle state: PENDING, state message: Installing libraries
renardeinside commented 3 years ago

hi @kelvin1794 , the error above happens when you launch the job a second time on the interactive cluster?

pebabion commented 3 years ago

@renardeinside yes, that's right.

But I have found a workaround: add 2 more step in GitHub actions workflow - first uninstall all the libraries on the cluster, then second restart the cluster.

After that, it can install the package just fine.

renardeinside commented 3 years ago

hi @kelvin1794,

Unfortunately, this is not directly related to dbx or cicd-templates, it's more of a way how cluster libraries are working with updates (it's not possible to reinstall the same library). I would recommend using dbx execute on interactive clusters, and use dbx launch on automated ones.

pebabion commented 3 years ago

Thank you, @renardeinside