artefactory / one-click-mlflow

A tool to deploy a mostly serverless MLflow tracking server on a GCP project with one command
GNU Lesser General Public License v3.0
65 stars 21 forks source link

MLflow app creation crashes because of protobuf version #82

Open ConsciousML opened 1 year ago

ConsciousML commented 1 year ago

Hi there,

Describe the bug When running make one-click-mlflow, an error appears while creating the app-engine-flex:

Error: Error waiting to create FlexibleAppVersion: Error waiting for Creating FlexibleAppVersion: Error code 9, message: An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>2023-03-28T10:26:33.695Z25166.wd.0: Traceback (most recent call last):
│   File "/usr/local/bin/mlflow", line 5, in <module>
│     from mlflow.cli import cli
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/__init__.py", line 32, in <module>
│     import mlflow.tracking._model_registry.fluent
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/__init__.py", line 8, in <module>
│     from mlflow.tracking.client import MlflowClient
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/client.py", line 8, in <module>
│     from mlflow.entities import ViewType
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/__init__.py", line 6, in <module>
│     from mlflow.entities.experiment import Experiment
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment.py", line 2, in <module>
│     from mlflow.entities.experiment_tag import ExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
│     from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/service_pb2.py", line 18, in <module>
│     from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 35, in <module>
│     serialized_options=None, file=DESCRIPTOR)
│   File "/usr/local/lib/python3.7/dist-packages/google/protobuf/descriptor.py", line 561, in __new__
│     _message.Message._CheckCalledFromGeneratedFile()
│ TypeError: Descriptors cannot not be created directly.
│ If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
│ If you cannot immediately regenerate your protos, some other possible workarounds are:
│  1. Downgrade the protobuf package to 3.20.x or lower.
│  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
│ 
│ More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
│ Traceback (most recent call last):
│   File "/usr/local/bin/mlflow", line 5, in <module>
│     from mlflow.cli import cli
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/__init__.py", line 32, in <module>
│     import mlflow.tracking._model_registry.fluent
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/__init__.py", line 8, in <module>
│     from mlflow.tracking.client import MlflowClient
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/tracking/client.py", line 8, in <module>
│     from mlflow.entities import ViewType
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/__init__.py", line 6, in <module>
│     from mlflow.entities.experiment import Experiment
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment.py", line 2, in <module>
│     from mlflow.entities.experiment_tag import ExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/entities/experiment_tag.py", line 2, in <module>
│     from mlflow.protos.service_pb2 import ExperimentTag as ProtoExperimentTag
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/service_pb2.py", line 18, in <module>
│     from .scalapb import scalapb_pb2 as scalapb_dot_scalapb__pb2
│   File "/usr/local/lib/python3.7/dist-packages/mlflow/protos/scalapb/scalapb_pb2.py", line 35, in <module>
│     serialized_options=None, file=DESCRIPTOR)
│   File "/usr/local/lib/python3.7/dist-packages/google/protobuf/descriptor.py", line 561, in __new__
│     _message.Message._CheckCalledFromGeneratedFile()
│ TypeError: Descriptors cannot not be created directly.
│ If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
│ If you cannot immediately regenerate your protos, some other possible workarounds are:
│  1. Downgrade the protobuf package to 3.20.x or lower.
│  2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).
│ 
│ More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates
│ 
│ 
│   with module.mlflow.module.server.google_app_engine_flexible_app_version.mlflow_app,
│   on modules/mlflow/server/main.tf line 112, in resource "google_app_engine_flexible_app_version" "mlflow_app":
│  112: resource "google_app_engine_flexible_app_version" "mlflow_app" {
│ 
╵

To Reproduce Steps to reproduce the behavior:

terraform version Terraform v1.4.2 on linux_amd64

  1. Install Terraform v1.4.2
  2. run make one-click-mlflow
  3. See error

Expected behavior The Terraform script should create the MLflow app without error

Desktop (please complete the following information):

ConsciousML commented 1 year ago

It works after forcing the protobuf version to the tracking_server\requirements.txt:

protobuf==3.20.0

Should I PR the change ?