search

mlflow / mlflow-export-import

Apache License 2.0
132 stars 78 forks source link

HTTP status code: 400. Reason: Bad Request when importing experiments from Legacy workspace into new #35

Closed tristansirrico closed 1 year ago

tristansirrico commented 2 years ago

Receiving errors importing experiments on a new, blank E2 workspace. The old, legacy workspace was exported successfully, but errors such as the following appear while importing:

Creating Databricks workspace directory '/Users/redacted@website.com/example'
Traceback (most recent call last):
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/bulk/import_experiments.py", line 15, in _import_experiment
    importer.import_experiment(exp_name, exp_input_dir)
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/experiment/import_experiment.py", line 35, in import_experiment
    mlflow_utils.set_experiment(self.mlflow_client, self.dbx_client, exp_name)
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/common/mlflow_utils.py", line 56, in set_experiment
    create_workspace_dir(dbx_client, os.path.dirname(exp_name))
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/common/mlflow_utils.py", line 97, in create_workspace_dir
    dbx_client.post("workspace/mkdirs", { "path": workspace_dir })
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/common/http_client.py", line 50, in post
    return json.loads(self._post(resource, data).text)
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/common/http_client.py", line 46, in _post
    self._check_response(rsp,uri)
  File "/home/ec2-user/mlflow-export-import/lib64/python3.7/site-packages/mlflow_export_import/common/http_client.py", line 63, in _check_response
    raise MlflowExportImportException(f"HTTP status code: {rsp.status_code}. Reason: {rsp.reason}. URI: {uri}. Params: {params}.")
mlflow_export_import.common.MlflowExportImportException: HTTP status code: 400. Reason: Bad Request. URI: https://my-workspace.cloud.databricks.com/api/2.0/workspace/mkdirs. Params: None.

Any way to see what the request is to determine why it is getting a 400? Could this be because the workspace is blank and the users do not exist yet?

amesar commented 2 years ago

The user associated with the Databricks PAT (Personal Access Token) you are using for importing must already exist. If you specify a bad PAT you should get a 403.

tristansirrico commented 2 years ago

The user associated with the Databricks PAT (Personal Access Token) you are using for importing must already exist. If you specify a bad PAT you should get a 403.

Hi @amesar, thanks for the response - I should have clarified that it is an almost blank workspace besides me creating an Admin user and generating a PAT for import use in the new account/workspace.

Also, I have been able to import some models fine, just experiments are giving trouble.

amesar commented 2 years ago

Hard to diagnose without the export dump. Can you send a zipped file of the export directory to andre at databricks.com?