Open smithj51 opened 4 months ago
Happens in python 3.9
google-api-core==2.12.0
google-api-python-client==2.100.0
google-auth==2.23.3
google-auth-httplib2==0.1.1
google-cloud-appengine-logging==1.1.2
google-cloud-audit-log==0.2.2
google-cloud-core==2.3.1
google-cloud-firestore==2.12.0
google-cloud-logging==3.10.0
...
proto-plus==1.22.3
protobuf==3.20.3
also.
Edit Seems like datadog-init issues?
/dd_tracer/python/bin/ddtrace-run gunicorn ...
works fine, but
/app/datadog-init /dd_tracer/python/bin/ddtrace-run gunicorn ...
Doesn't
Edit 2: datadog-init internally use protobuf-5.27.0, which was causing the issue.
@yhs0602 How did you fix that issue with datadog-init? I am pinning protobuf==3.20.3 and still getting issues
@yhs0602 how did you solve this?
@doudoujay This is my solution although this seems risky to be deployed to a production environment.
So I found the issue by comparing an old version of my working docker image and new one. Inside my broken container:
(base) ➜ ~ broken
root@288a9552d8c1:/api# pip show protobuf
Name: protobuf
Version: 4.25.3
Summary:
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /usr/local/lib/python3.10/site-packages
Requires:
Required-by: ddtrace, google-api-core, google-cloud-appengine-logging, google-cloud-audit-log, google-cloud-logging, googleapis-common-protos, grpc-google-iam-v1, grpcio-status, proto-plus
root@288a9552d8c1:/api# /app/datadog-init pip show protobuf
Name: protobuf
Version: 5.27.0
Summary:
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /dd_tracer/python
Requires:
Required-by: ddtrace, google-api-core, google-cloud-appengine-logging, google-cloud-audit-log, google-cloud-logging, googleapis-common-protos, grpc-google-iam-v1, grpcio-status, proto-plus
Inside my old working container:
(base) ➜ Downloads working
root@b5987abfa0e3:/api# pip show protobuf
Name: protobuf
Version: 4.25.3
Summary:
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /usr/local/lib/python3.10/site-packages
Requires:
Required-by: ddtrace, google-api-core, google-cloud-appengine-logging, google-cloud-audit-log, google-cloud-logging, googleapis-common-protos, grpc-google-iam-v1, grpcio-status, proto-plus
root@b5987abfa0e3:/api# /app/datadog-init pip show protobuf
Name: protobuf
Version: 5.26.1
Summary:
Home-page: https://developers.google.com/protocol-buffers/
Author: protobuf@googlegroups.com
Author-email: protobuf@googlegroups.com
License: 3-Clause BSD License
Location: /dd_tracer/python
Requires:
Required-by: ddtrace, google-api-core, google-cloud-appengine-logging, google-cloud-audit-log, google-cloud-logging, googleapis-common-protos, grpc-google-iam-v1, grpcio-status, proto-plus
Basically datadog-init is bundling it's own python dependencies without pinning them strongly so when the protobuf==5.27.0 was released this causing a conflict. In reality the lower versions of protobuf are not supposed to work with the high versions at all they only happened to be compatible it seems to me.
So my original docker image looked like this:
FROM python:3.10.8
COPY . .
RUN pip install -r requirements.txt
<other random dependencies>
# Prep for datadog
COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
# set up a directory to run out of
WORKDIR /api
# Expose the running port
EXPOSE 5150
# set the app we want to run
ENV FLASK_APP=app.py
ENV LOGGING='formatted'
# Move app over
COPY . .
RUN chmod +x /api/entrypoint.sh
ENTRYPOINT ["/app/datadog-init", "/api/entrypoint.sh"]
My solution docker image looks like this:
# Use python:3.10.8 as the builder
FROM python:3.10.8 as builder
# Prep for datadog
COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ ddtrace
RUN /app/datadog-init pip uninstall -y protobuf==5.27.0
RUN /app/datadog-init python -m pip install --force-reinstall --ignore-installed protobuf==5.26.1
# Use the base image
FROM <base image where I install other pip dependencies>
# Copy datadog-init and resolved dependencies from the builder
COPY --from=builder /app/datadog-init /app/datadog-init
COPY --from=builder /dd_tracer /dd_tracer
# set up a directory to run out of
WORKDIR /api
# Expose the running port
EXPOSE 5150
# set the app we want to run
ENV FLASK_APP=app.py
ENV LOGGING='formatted'
WORKDIR /api/dockerbuild
RUN python CustomCerts.py
# Move back to the api directory
WORKDIR /api
# Move app over
COPY . .
RUN chmod +x /api/entrypoint.sh
ENTRYPOINT ["/app/datadog-init"]
CMD ["/dd_tracer/python/bin/ddtrace-run", "/api/entrypoint.sh"]
my simple solution --
proto-plus==1.24.0.dev1
:)
ddtrace relies on any protobuf >=3 The solution that we adopted was to specify in protobuf version before installing ddtrace.
COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ protobuf==5.26.1 ddtrace
A similar approach we took for the case when we had another dependencies that relies even on older protobuf version (mlflow
)
COPY --from=datadog/serverless-init:1 /datadog-init /app/datadog-init
RUN pip install --target /dd_tracer/python/ protobuf==4.25.3 ddtrace
It looks like python proto plus just submitted a change to fix compatibility. Can y'all check to see if that fixes your issue?
It looks like python proto plus just submitted a change to fix compatibility. Can y'all check to see if that fixes your issue?
Yes that fixed it for me using the 1.24.0.dev1 release
It looks like python proto plus just submitted a change to fix compatibility. Can y'all check to see if that fixes your issue?
thank you, i was able to resolve the issue
@deannagarcia, This issue can be closed now as a formal release of proto-plus
was published 2 weeks ago: https://pypi.org/project/proto-plus/1.24.0/
Still have the issue with upgraded libraries
(no error)
google-api-core-2.19.1 google-api-python-client-2.136.0 google-auth-2.31.0 google-auth-httplib2-0.2.0 google-cloud-resource-manager-1.12.3 googleapis-common-protos-1.63.2 grpc-google-iam-v1-0.13.1 grpcio-1.64.1 grpcio-status-1.62.2 httplib2-0.22.0 idna-3.7 proto-plus-1.24.0 protobuf-4.25.3
(error)
google-api-core-2.19.1 google-api-python-client-2.137.0 google-auth-2.32.0 google-auth-httplib2-0.2.0 google-cloud-resource-manager-1.12.4 googleapis-common-protos-1.63.2 grpc-google-iam-v1-0.13.1 grpcio-1.64.1 grpcio-status-1.64.1 httplib2-0.22.0 idna-3.7 proto-plus-1.24.0 protobuf-5.27.2
@anilkumarmyla, Please could you share the full stack trace?
Please also share the output of
>>> import proto
>>> proto.__version__
@parthea
bash-3.2$ python3 --version
Python 3.12.4
bash-3.2$ python3 -m venv .venv
bash-3.2$ source .venv/bin/activate
(.venv) bash-3.2$ gcloud --verbosity=debug --version
DEBUG: Running [gcloud.version] with arguments: []
DEBUG: Chosen display Format:flattened[no-pad,separator=" "]
INFO: Display format: "flattened[no-pad,separator=" "]"
Google Cloud SDK 483.0.0
alpha 2024.06.28
bq 2.1.6
core 2024.06.28
gcloud-crc32c 1.0.0
gsutil 5.30
(.venv) bash-3.2$ pip3 install proto-plus
Collecting proto-plus
Using cached proto_plus-1.24.0-py3-none-any.whl.metadata (2.2 kB)
Collecting protobuf<6.0.0dev,>=3.19.0 (from proto-plus)
Using cached protobuf-5.27.2-cp38-abi3-macosx_10_9_universal2.whl.metadata (592 bytes)
Using cached proto_plus-1.24.0-py3-none-any.whl (50 kB)
Using cached protobuf-5.27.2-cp38-abi3-macosx_10_9_universal2.whl (412 kB)
Installing collected packages: protobuf, proto-plus
Successfully installed proto-plus-1.24.0 protobuf-5.27.2
[notice] A new release of pip is available: 24.0 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(.venv) bash-3.2$ gcloud --verbosity=debug --version
DEBUG: Running [gcloud.version] with arguments: []
DEBUG: Chosen display Format:flattened[no-pad,separator=" "]
INFO: Display format: "flattened[no-pad,separator=" "]"
DEBUG: module 'google._upb._message' has no attribute 'MessageMapContainer'
Traceback (most recent call last):
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
resources = calliope_command.Run(cli=self, args=args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 817, in Run
display_info=self.ai.display_info).Display()
^^^^^^^^^
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/calliope/display.py", line 487, in Display
self._printer.Print(self._resources)
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_printer_base.py", line 299, in Print
self.AddRecord(resources, delimit=intermediate)
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_printer_base.py", line 239, in AddRecord
self._AddRecord(self._process_record(record), delimit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_projector.py", line 517, in Evaluate
return self._Project(obj, self._projection.Tree(), flag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_projector.py", line 433, in _Project
import proto # pylint: disable=g-import-not-at-top
^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/__init__.py", line 15, in <module>
from .enums import Enum
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/enums.py", line 19, in <module>
from proto import _file_info
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/_file_info.py", line 24, in <module>
from proto.marshal.rules.message import MessageRule
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/marshal/__init__.py", line 15, in <module>
from .marshal import Marshal
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/marshal/marshal.py", line 25, in <module>
from proto.marshal import compat
File "/opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto/marshal/compat.py", line 43, in <module>
map_composite_types += (_message.MessageMapContainer,)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'google._upb._message' has no attribute 'MessageMapContainer'
ERROR: gcloud crashed (AttributeError): module 'google._upb._message' has no attribute 'MessageMapContainer'
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
(.venv) bash-3.2$ pip3 install protobuf==5.26.1
Collecting protobuf==5.26.1
Downloading protobuf-5.26.1-cp37-abi3-macosx_10_9_universal2.whl.metadata (592 bytes)
Downloading protobuf-5.26.1-cp37-abi3-macosx_10_9_universal2.whl (404 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 404.0/404.0 kB 2.7 MB/s eta 0:00:00
Installing collected packages: protobuf
Attempting uninstall: protobuf
Found existing installation: protobuf 5.27.2
Uninstalling protobuf-5.27.2:
Successfully uninstalled protobuf-5.27.2
Successfully installed protobuf-5.26.1
[notice] A new release of pip is available: 24.0 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(.venv) bash-3.2$ gcloud --verbosity=debug --version
DEBUG: Running [gcloud.version] with arguments: []
DEBUG: Chosen display Format:flattened[no-pad,separator=" "]
INFO: Display format: "flattened[no-pad,separator=" "]"
Google Cloud SDK 483.0.0
alpha 2024.06.28
bq 2.1.6
core 2024.06.28
gcloud-crc32c 1.0.0
gsutil 5.30
(.venv) bash-3.2$ python3
Python 3.12.4 (main, Jun 6 2024, 18:26:44) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import proto
>>> proto.__version__
'1.24.0'
>>>
Thanks for sharing the stack trace @anilkumarmyla! The stack trace is slightly different from the one in https://github.com/protocolbuffers/protobuf/issues/16935#issue-2313399899 in that the error originates from proto-plus
via the gcloud
library in /opt/homebrew/Caskroom/google-cloud-sdk/481.0.0/google-cloud-sdk/lib/third_party/proto
, rather than the installed version. This issue is being investigated. (Googlers see internal issue b/352040706)
@parthea i'm able to reproduce the same using a docker container that has google-cloud-cli rpm package installed. In this setup I'm unable to reproduce if i'm using a venv, but can reproduce on the base system as below
[root@da99b97f53ef /]# python3 --version
Python 3.11.3
[root@da99b97f53ef /]# gcloud --verbosity=debug --version
DEBUG: Running [gcloud.version] with arguments: []
DEBUG: Chosen display Format:flattened[no-pad,separator=" "]
INFO: Display format: "flattened[no-pad,separator=" "]"
Google Cloud SDK 481.0.0
alpha 2024.06.14
beta 2024.06.14
bq 2.1.6
bundled-python3-unix 3.11.8
core 2024.06.14
gcloud-crc32c 1.0.0
gsutil 5.30
DEBUG: SDK update checks are disabled.
[root@da99b97f53ef /]# pip3 install --user proto-plus
Installing collected packages: protobuf, proto-plus
Successfully installed proto-plus-1.24.0 protobuf-5.27.2
[root@da99b97f53ef /]# gcloud --verbosity=debug --version
DEBUG: Running [gcloud.version] with arguments: []
DEBUG: Chosen display Format:flattened[no-pad,separator=" "]
INFO: Display format: "flattened[no-pad,separator=" "]"
DEBUG: module 'google._upb._message' has no attribute 'MessageMapContainer'
Traceback (most recent call last):
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 998, in Execute
resources = calliope_command.Run(cli=self, args=args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/backend.py", line 817, in Run
display_info=self.ai.display_info).Display()
^^^^^^^^^
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/calliope/display.py", line 487, in Display
self._printer.Print(self._resources)
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_printer_base.py", line 299, in Print
self.AddRecord(resources, delimit=intermediate)
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_printer_base.py", line 239, in AddRecord
self._AddRecord(self._process_record(record), delimit)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_projector.py", line 517, in Evaluate
return self._Project(obj, self._projection.Tree(), flag)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/google-cloud-sdk/lib/googlecloudsdk/core/resource/resource_projector.py", line 433, in _Project
import proto # pylint: disable=g-import-not-at-top
^^^^^^^^^^^^
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/__init__.py", line 15, in <module>
from .enums import Enum
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/enums.py", line 19, in <module>
from proto import _file_info
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/_file_info.py", line 24, in <module>
from proto.marshal.rules.message import MessageRule
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/marshal/__init__.py", line 15, in <module>
from .marshal import Marshal
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/marshal/marshal.py", line 25, in <module>
from proto.marshal import compat
File "/usr/lib64/google-cloud-sdk/lib/third_party/proto/marshal/compat.py", line 43, in <module>
map_composite_types += (_message.MessageMapContainer,)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'google._upb._message' has no attribute 'MessageMapContainer'
ERROR: gcloud crashed (AttributeError): module 'google._upb._message' has no attribute 'MessageMapContainer'
If you would like to report this issue, please run the following command:
gcloud feedback
To check gcloud for common problems, please run the following command:
gcloud info --run-diagnostics
A fix was merged in proto-plus
which handles the AttributeError
. I expect this fix will make it into the next gcloud CLI release.
@parthea thanks for the fix, do let us know when it makes it into the gcloud release
I have a python script based on Gemini API SDK working ok till yesterday 2024-09-22 Today: The "import google.generativeai as genai" results in the "Attribute error...." My environment.
@parthea I did your suggestion:
import proto This command ends up like this: .... from proto.marshal import compat
File ~\AppData\Local\anaconda3\envs\llm-poc\Lib\site-packages\proto\marshal\compat.py:60 map_composite_types += (_message.MessageMapContainer,)
AttributeError: module 'google._upb._message' has no attribute 'MessageMapContainer'
Any short term fix to allow the project development to resume.
Thank you.
Benoit
What version of protobuf and what language are you using? Version: v27.0 Language: Python
What operating system (Linux, Windows, ...) and version? Windows
What runtime / compiler are you using (e.g., python version or gcc version) python in gitlab ci/cd
What did you do? Steps to reproduce the behavior: python pip install following packages: MarkupSafe, werkzeug, click, pathspec, logbook, protobuf, pytz, Babel, text-unidecode, python-slugify, leather, six, isodate, pytimeparse, future, parsedatetime, agate, pkgutil-resolve-name, attrs, rpds-py, referencing, zipp, importlib-resources, jsonschema-specifications, jsonschema, python-dateutil, hologram, pycparser, cffi, charset-normalizer, certifi, urllib3, idna, requests, packaging, Jinja2, typing-extensions, msgpack, mashumaro, minimal-snowplow-tracker, colorama, networkx, dbt-extractor, pyyaml, sqlparse, dbt-core, googleapis-common-protos, proto-plus, pyasn1, rsa, pyasn1-modules, cachetools, google-auth, google-api-core, google-crc32c, google-cloud-core, google-resumable-media, google-cloud-storage, grpcio, grpc-google-iam-v1, google-cloud-dataproc, google-cloud-bigquery, dbt-bigquery, portalocker, platformdirs, tblib, importlib-metadata, distlib, filelock, virtualenv, isolate, isolate-proto, structlog, types-python-dateutil, grpc-interceptor, frozenlist, async-timeout, multidict, yarl, aiosignal, aiohttp, pyjwt, cryptography, auth0-python, opentelemetry-semantic-conventions, wrapt, deprecated, opentelemetry-api, opentelemetry-sdk, pydantic, datadog-api-client, dill, mdurl, markdown-it-py, pygments, rich, sniffio, h11, exceptiongroup, anyio, httpcore, httpx, starlette, fastapi, fal, backports.functools-lru-cache, backoff, monotonic, posthog, greenlet, sqlalchemy, deprecation, astor, agate-sql, numpy, pandas, dbt-fal, slack-sdk, tabulate, pyarrow, db-dtypes
What did you expect to see compatibility with google APIs
What did you see instead?
Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs). Full error log from gitlab ci/cd job log:
Anything else we should know about your project / environment using docker image python:3.8.1-slim-buster