search

googleapis / python-aiplatform

A Python SDK for Vertex AI, a fully managed, end-to-end platform for data science and machine learning.
Apache License 2.0
615 stars 328 forks source link

Installing python-aiplatform && KFP on the same enviroment breaks KFP #2921

Open Davidnet opened 10 months ago

Davidnet commented 10 months ago

Environment details

Steps to reproduce

  1. Create a new venv
  2. `python -m pip install kfp==2.4.0
  3. python -m pip install google-cloud-aiplatform==1.36.0

Code example

import kfp

Stack trace

Traceback (most recent call last):
  File "/home/davidnet/Documents/personal/aip_problem/main.py", line 1, in <module>
    import kfp
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/__init__.py", line 40, in <module>
    from kfp import components  # noqa: keep unused import
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/components/__init__.py", line 27, in <module>
    from kfp.components.load_yaml_utilities import load_component_from_file
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/components/load_yaml_utilities.py", line 18, in <module>
    from kfp.dsl import structures
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/__init__.py", line 232, in <module>
    from kfp.dsl.component_decorator import component
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/component_decorator.py", line 19, in <module>
    from kfp.dsl import component_factory
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/component_factory.py", line 27, in <module>
    from kfp.dsl import container_component_class
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/container_component_class.py", line 18, in <module>
    from kfp.dsl import base_component
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/base_component.py", line 19, in <module>
    from kfp.dsl import pipeline_task
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/pipeline_task.py", line 26, in <module>
    from kfp.dsl import structures
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/dsl/structures.py", line 34, in <module>
    from kfp.pipeline_spec import pipeline_spec_pb2
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/pipeline_spec/pipeline_spec_pb2.py", line 38, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/google/protobuf/descriptor.py", line 780, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

The only way to solve it is to set the enviroment variable, should this be also filled in kfp?

Thanks

Ark-kun commented 10 months ago

Looks like there is a protobuf compat issue causing problems with autogenerated code:

  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/kfp/pipeline_spec/pipeline_spec_pb2.py", line 38, in <module>
    _descriptor.EnumValueDescriptor(
  File "/home/davidnet/Documents/personal/aip_problem/.venv/lib/python3.11/site-packages/google/protobuf/descriptor.py", line 780, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).