search

mlflow / mlflow

Open source platform for the machine learning lifecycle
https://mlflow.org
Apache License 2.0
18.01k stars 4.07k forks source link

[BUG] 413 Client Error: Request Entity Too Large for url #10332

Open dotdothu opened 8 months ago

dotdothu commented 8 months ago

Issues Policy acknowledgement

Where did you encounter this bug?

Other

Willingness to contribute

Yes. I can contribute a fix for this bug independently.

MLflow version

System information

Describe the problem

When I try to log a model above a certain size (25MB worked, probably above 50?) then I face an error.

I use Google Cloud Run, PostgreSQL and Google Cloud Storage

MLFLOW_GCS_UPLOAD_CHUNK_SIZE set for 262144

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>413 Request Entity Too Large</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Request Entity Too Large</h1>
<h2>Your client issued a request that was too large.
</h2>

<h2></h2>
</body></html>

Tracking information

MLflow version: 2.7.1
Tracking URI: https://X/
Artifact URI: mlflow-artifacts:/77/cf85568b0464465dab072417b98eb3bf/artifacts
System information: Windows 10.0.19045
Python version: 3.11.4
MLflow version: 2.7.1
Tracking URI: https://X/
Registry URI: https://X/
Active experiment ID: 77
Active run ID: cf85568b0464465dab072417b98eb3bf
Active run artifact URI: mlflow-artifacts:/77/cf85568b0464465dab072417b98eb3bf/artifacts
MLflow dependencies: 
  Flask: 2.3.3
  Jinja2: 3.1.2
  alembic: 1.12.0
  click: 8.1.7
  cloudpickle: 2.2.1
  databricks-cli: 0.18.0
  docker: 6.1.3
  entrypoints: 0.4
  gitpython: 3.1.40
  google-cloud-storage: 2.12.0
  importlib-metadata: 6.8.0
  markdown: 3.5
  matplotlib: 3.8.0
  numpy: 1.26.1
  packaging: 23.2
  pandas: 2.1.1
  protobuf: 4.24.4
  psutil: 5.9.6
  pyarrow: 13.0.0
  pytz: 2023.3.post1
  pyyaml: 6.0.1
  querystring-parser: 1.2.4
  requests: 2.31.0
  scikit-learn: 1.3.2
  scipy: 1.11.3
  sqlalchemy: 2.0.22
  sqlparse: 0.4.4
  waitress: 2.1.2

Code to reproduce issue

# Works
rf = RandomForestRegressor(n_estimators=10, random_state=42)
rf.fit(X_train, y_train)
signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

# Fails
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

Stack trace

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\utils\request_utils.py:32, in augmented_raise_for_status(response)
     [31](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:31) try:
---> [32](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:32)     response.raise_for_status()
     [33](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:33) except HTTPError as e:

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\.venv\Lib\site-packages\requests\models.py:1021, in Response.raise_for_status(self)
   [1020](file:///C:/Users/ML/repositories/mlflow-guide-demo/.venv/Lib/site-packages/requests/models.py:1020) if http_error_msg:
-> [1021](file:///C:/Users/ML/repositories/mlflow-guide-demo/.venv/Lib/site-packages/requests/models.py:1021)     raise HTTPError(http_error_msg, response=self)

HTTPError: 413 Client Error: Request Entity Too Large for url: https://ada-mlflow-preprd-juqzwmg2lq-nw.a.run.app/api/2.0/mlflow-artifacts/artifacts/77/30a5817a7c9d4ffe9a50481d7816c9c2/artifacts/models/model.pkl

During handling of the above exception, another exception occurred:

HTTPError                                 Traceback (most recent call last)
[c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\01_experiment_tracking.ipynb Cell 16 line 2
      [1](vscode-notebook-cell:/c%3A/Users/ML/repositories/mlflow-guide-demo/demo/01_experiment_tracking.ipynb#X51sZmlsZQ%3D%3D?line=0) signature = mlflow.models.infer_signature(X_test, rf.predict(X_test))
----> [2](vscode-notebook-cell:/c%3A/Users/ML/repositories/mlflow-guide-demo/demo/01_experiment_tracking.ipynb#X51sZmlsZQ%3D%3D?line=1) mlflow.sklearn.log_model(rf, artifact_path=model_folder_name, signature=signature)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:408, in log_model(sk_model, artifact_path, conda_env, code_paths, serialization_format, registered_model_name, signature, input_example, await_registration_for, pip_requirements, extra_pip_requirements, pyfunc_predict_fn, metadata)
    [328](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:328) @format_docstring(LOG_MODEL_PARAM_DOCS.format(package_name="scikit-learn"))
    [329](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:329) def log_model(
    [330](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:330)     sk_model,
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:1'>1</a>;32m   (...)
    [342](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:342)     metadata=None,
    [343](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:343) ):
    [344](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:344)     """
    [345](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:345)     Log a scikit-learn model as an MLflow artifact for the current run. Produces an MLflow Model
    [346](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:346)     containing the following flavors:
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\sklearn\__init__.py:1'>1</a>;32m   (...)
    [406](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:406) 
    [407](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:407)     """
--> [408](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:408)     return Model.log(
    [409](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:409)         artifact_path=artifact_path,
    [410](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:410)         flavor=mlflow.sklearn,
    [411](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:411)         sk_model=sk_model,
    [412](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:412)         conda_env=conda_env,
    [413](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:413)         code_paths=code_paths,
    [414](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:414)         serialization_format=serialization_format,
    [415](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:415)         registered_model_name=registered_model_name,
    [416](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:416)         signature=signature,
    [417](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:417)         input_example=input_example,
    [418](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:418)         await_registration_for=await_registration_for,
    [419](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:419)         pip_requirements=pip_requirements,
    [420](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:420)         extra_pip_requirements=extra_pip_requirements,
    [421](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:421)         pyfunc_predict_fn=pyfunc_predict_fn,
    [422](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:422)         metadata=metadata,
    [423](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/sklearn/__init__.py:423)     )

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\models\model.py:580, in Model.log(cls, artifact_path, flavor, registered_model_name, await_registration_for, metadata, **kwargs)
    [578](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:578)     _logger.warning(_LOG_MODEL_MISSING_SIGNATURE_WARNING)
    [579](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:579) flavor.save_model(path=local_path, mlflow_model=mlflow_model, **kwargs)
--> [580](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:580) mlflow.tracking.fluent.log_artifacts(local_path, mlflow_model.artifact_path)
    [581](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:581) try:
    [582](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/models/model.py:582)     mlflow.tracking.fluent._record_logged_model(mlflow_model)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\fluent.py:908, in log_artifacts(local_dir, artifact_path)
    [878](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:878) """
    [879](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:879) Log all the contents of a local directory as artifacts of the run. If no run is active,
    [880](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:880) this method will create a new active run.
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\tracking\fluent.py:1'>1</a>;32m   (...)
    [905](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:905)         mlflow.log_artifacts("data", artifact_path="states")
    [906](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:906) """
    [907](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:907) run_id = _get_or_start_run().info.run_id
--> [908](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/fluent.py:908) MlflowClient().log_artifacts(run_id, local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\client.py:1138, in MlflowClient.log_artifacts(self, run_id, local_dir, artifact_path)
   [1094](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1094) def log_artifacts(
   [1095](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1095)     self, run_id: str, local_dir: str, artifact_path: Optional[str] = None
   [1096](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1096) ) -> None:
   [1097](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1097)     """
   [1098](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1098)     Write a directory of files to the remote ``artifact_uri``.
   [1099](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1099) 
ref='c:\Users\ML\OneDrive\repositories\mlflow-guide-demo\demo\mlflow\tracking\client.py:1'>1</a>;32m   (...)
   [1136](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1136)         is_dir: True
   [1137](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1137)     """
-> [1138](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/client.py:1138)     self._tracking_client.log_artifacts(run_id, local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\tracking\_tracking_service\client.py:463, in TrackingServiceClient.log_artifacts(self, run_id, local_dir, artifact_path)
    [456](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:456) def log_artifacts(self, run_id, local_dir, artifact_path=None):
    [457](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:457)     """
    [458](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:458)     Write a directory of files to the remote ``artifact_uri``.
    [459](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:459) 
    [460](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:460)     :param local_dir: Path to the directory of files to write.
    [461](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:461)     :param artifact_path: If provided, the directory in ``artifact_uri`` to write to.
    [462](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:462)     """
--> [463](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/tracking/_tracking_service/client.py:463)     self._get_artifact_repo(run_id).log_artifacts(local_dir, artifact_path)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\store\artifact\http_artifact_repo.py:45, in HttpArtifactRepository.log_artifacts(self, local_dir, artifact_path)
     [41](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:41)     artifact_dir = (
     [42](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:42)         posixpath.join(artifact_path, rel_path) if artifact_path else rel_path
     [43](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:43)     )
     [44](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:44) for f in filenames:
---> [45](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:45)     self.log_artifact(os.path.join(root, f), artifact_dir)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\store\artifact\http_artifact_repo.py:31, in HttpArtifactRepository.log_artifact(self, local_file, artifact_path)
     [27](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:27) with open(local_file, "rb") as f:
     [28](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:28)     resp = http_request(
     [29](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:29)         self._host_creds, endpoint, "PUT", data=f, extra_headers=extra_headers
     [30](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:30)     )
---> [31](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/store/artifact/http_artifact_repo.py:31)     augmented_raise_for_status(resp)

File [c:\Users\ML\OneDrive](file:///C:/Users/ML)\repositories\mlflow-guide-demo\demo\mlflow\utils\request_utils.py:35, in augmented_raise_for_status(response)
     [33](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:33) except HTTPError as e:
     [34](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:34)     if response.text:
---> [35](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:35)         raise HTTPError(
     [36](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:36)             f"{e}. Response text: {response.text}", request=e.request, response=e.response
     [37](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:37)         )
     [38](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:38)     else:
     [39](file:///C:/Users/ML/repositories/mlflow-guide-demo/demo/mlflow/utils/request_utils.py:39)         raise e

HTTPError: 413 Client Error: Request Entity Too Large for url: https://ada-mlflow-preprd-juqzwmg2lq-nw.a.run.app/api/2.0/mlflow-artifacts/artifacts/77/30a5817a7c9d4ffe9a50481d7816c9c2/artifacts/models/model.pkl. Response text: 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>413 Request Entity Too Large</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Request Entity Too Large</h1>
<h2>Your client issued a request that was too large.
</h2>
<h2></h2>
</body></html>

Other info / logs

REPLACE_ME

What component(s) does this bug affect?

What interface(s) does this bug affect?

What language(s) does this bug affect?

What integration(s) does this bug affect?

github-actions[bot] commented 8 months ago

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

AwaisPF commented 3 months ago

any update on this? because i am facing same issue mlflow version 2.11.1 same version on both client and server

viktorandreevichmorozov commented 3 months ago

the same for me

cile98 commented 3 months ago

I had previously encountered the same issue. In my case it was caused by proxy-body-size nginx parameter. I've set it to 256m and it fixed the problem (sizes of my models were around 40-50 MB)