search

databrickslabs / cicd-templates

Manage your Databricks deployments and CI with code.
Other
201 stars 100 forks source link

Problem executing my jobs to cluster dbx deploy --jobs=project_name fails #80

Open echarso opened 2 years ago

echarso commented 2 years ago

I am keep getting this error when using this project.

response = verify_rest_response(response, endpoint) File "/Users/charalampossouris/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 137, in verify_rest_response raise MlflowException("%s. Response body: '%s'" % (base_msg, response.text)) mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html>Databricks - Sign In

'

I am almost confident that my DATABRICKS-HOST and DATABRICKS-TOKEN are correct and placed in the right places.

I will continue debugging it but I was hoping for some hints :)

When executing "databricks workspace list" I can see the workspaces, which indicates to me that my credentials are correctly placed.

renardeinside commented 2 years ago

hi @echarso , I guess the root cause is in the host path - please take a look at this section.

echarso commented 2 years ago

Thanks @renardeinside for your answer. I just tried it and didn't work :( Should I have a cluster in my deployment file ? I am using the standard deployment file, the one provided when doing pip install

echarso commented 2 years ago

I am using Databricks in AWS and I have Mac M1...

renardeinside commented 2 years ago

Let's try to debug this on your local machine first.

Could you please provide the following:

Also, please provide the full error log, the truncated part unfortunately is not providing enough info.

echarso commented 2 years ago

Hi again! Thanks for taking time on this :) Databricks Version 0.16.2 DataBricks eXtensions aka dbx, version ~> 0.2.1 mlflow, version 1.14.1

I must admit that now it works from the githubs actions . Not locally though.

[dbx][2021-11-18 11:01:20.404] Starting new deployment for environment default [dbx][2021-11-18 11:01:20.404] Using configuration from the environment variables Traceback (most recent call last): File "/Users/-/miniforge3/bin/dbx", line 8, in sys.exit(cli()) File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/-/miniforge3/lib/python3.9/site-packages/click/core.py", line 610, in invoke return callback(args, **kwargs) File "/Users/-/miniforge3/lib/python3.9/site-packages/dbx/commands/deploy.py", line 122, in deploy api_client = prepare_environment(environment) File "/Users/-/miniforge3/lib/python3.9/site-packages/dbx/utils/common.py", line 334, in prepare_environment experiment: Optional[mlflow.entities.Experiment] = mlflow.get_experiment_by_name(environment_data["workspace_dir"]) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/fluent.py", line 795, in get_experiment_by_name return MlflowClient().get_experiment_by_name(name) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/client.py", line 419, in get_experiment_by_name return self._tracking_client.get_experiment_by_name(name) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py", line 128, in get_experiment_by_name return self.store.get_experiment_by_name(name) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 319, in get_experiment_by_name raise e File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 306, in get_experiment_by_name response_proto = self._call_endpoint(GetExperimentByName, req_body) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py", line 55, in _call_endpoint return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 170, in call_endpoint response = verify_rest_response(response, endpoint) File "/Users/-/miniforge3/lib/python3.9/site-packages/mlflow/utils/rest_utils.py", line 137, in verify_rest_response raise MlflowException("%s. Response body: '%s'" % (base_msg, response.text)) mlflow.exceptions.MlflowException: API request to endpoint was successful but the response body was not in a valid JSON format. Response body: '<!doctype html>Databricks - Sign In

'

renardeinside commented 2 years ago

hi @echarso , please try out the following:

  1. update to dbx 0.2.2
  2. verify that your provided hostname are in variables: DATABRICKS_HOST & DATABRICKS_TOKEN inside your CI pipeline and your local environment.