ImportError: symbol not found in flat namespace (protobuf 3.20.2)

[pietrodn]

What version of protobuf and what language are you using? Python, protobuf==3.20.2

What operating system (Linux, Windows, ...) and version? macOS 12.6

What runtime / compiler are you using (e.g., python version or gcc version) Python 3.10.6

What did you do?

python3 -m venv venv
. venv/bin/activate
pip install protobuf==3.20.2
python3 -c "import google.protobuf.descriptor"

What did you expect to see Nothing (a successful import)

What did you see instead?

>>> import google.protobuf.descriptor
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: dlopen(.venv/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

FYI a similar error is happening inside Google's google-cloud-bigquery library.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment

1. The issue does not happen in a Linux environment (our CI).
2. The issue does not happen with protobuf==3.20.1.
3. The issue does not happen on Python 3.8.

[jcrm1]

Also getting this error (still on macOS 12.6, Python 3.10.6)

[madsjk816]

+1 here, macOS 12.6, Python 3.10.6, protobuf==3.20.2 and Mac M1

[miohtama]

Same: macOS 12.6, Python 3.10.6, protobuf==3.20.2 and Mac M1.

For me, I am not using even using Protobuf, but one of the dependencies imports it on a startup and this side effect causes the codebase no longer run:

    from ethpm.dependencies import (
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/ethpm/dependencies.py", line 7, in <module>
    from ethpm.validation.package import (
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/ethpm/validation/package.py", line 11, in <module>
    from ethpm._utils.ipfs import (
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
    exec(co, module.__dict__)
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/ethpm/_utils/ipfs.py", line 18, in <module>
    from google.protobuf.descriptor import (
  File "/Users/moo/Library/Caches/pypoetry/virtualenvs/backend-WPuQnUi4-py3.10/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message

[miohtama]

Confirming downgrade works:

  • Updating protobuf (3.20.2 -> 3.20.1)

[cinnil]

Had this same issue pop up while installing lstein/stable-diffusion and downgrading solved it as well. Thanks!

Confirming downgrade works:

  • Updating protobuf (3.20.2 -> 3.20.1)

[AlexSchultz-clumio]

3.19.5 also suffers from the same issue. Downgrading to 3.19.4 works.

[nikiz24]

in my case brew install protobuf brew link --overwrite protobuf works. Others like downgrade pip protobuf does not work for me.

[nikiz24]

in my case brew install protobuf brew link --overwrite protobuf works. Others like downgrade pip protobuf does not work for me.

[scomma]

3.19.5 also suffers from the same issue. Downgrading to 3.19.4 works.

Confirmed fix, thanks! Came here trying to install stable diffuser.

[bh2smith]

Issue also doesn't happen on python 3.9!

[Ported]

Running "pip install protobuf==3.19.4" to downgrade also worked for me.

[pro0517]

+1 here, macOS 11.2, Python 3.10.7, protobuf==3.20.2 and Mac M1

[lightnessofbein]

I used to have troubles with protobuf==3.20.2 (ImportError cause by some symbols) right after import feast Downgrading to 3.20.1 solves the problem

[Roymprog]

I do experience the same problem with Python version 3.8.13 using poetry. Perhaps it's related to this issue. Neither version 3.20.1 nor 3.19.4 seems to work.

I have found that protobuf-3.20.1-py2.py3-none-any.whl fails, and protobuf-3.20.1-cp38-cp38-macosx_10_9_x86_64.whl works.

EDIT: Solved my issue by being explicit about how to build protobuf on M1 Mac: pip install protobuf==3.20 --install-option="--cpp_implementation"

[pietrodn]

There's a known security vulnerability that affects 3.20.1 and is fixed by 3.20.2. However, 3.20.2 is unusable because of this bug.

Google should deal with this with high priority.

[NicolasSompairac]

Simply came here to confirm that downgrading protobuf from 3.19.5 to 3.19.4 removed this error. I was getting this message when trying to use Stable Diffusion on my M2 Mac with Python 3.10.6.

[mayeut]

The issue is still there with 3.20.3 macOS 12.6, Python 3.10.7, protobuf==3.20.3 and Mac M1

[tommyjcarpenter]

For us, we didnt have to go to 3.19. but had to go protobuf (3.20.3 -> 3.20.1)

[pietrodn]

It appears that the same bug was introduced in parallel in 3.19.5 and 3.20.2, most likely as a result of backporting some bug fix to the earlier minor revision.

[deannagarcia]

Sorry about this issue. The problem is that the 3.19 and 3.20 branches only used the pure python implementation of protobuf. The newest update has added an incomplete C++ implementation for M1 binaries which has caused the crash.

To work around, you can set the environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python.

We are working on a fix now, but in order to get the most efficient python implementation we would recommend you upgrade to protobuf 4.21.7.

[deannagarcia]

I believe this issue should be fixed for 3.19.5, 3.19.6, 3.20.2, and 3.20.3. Please let me know if you are still experiencing issues.

The fix will force python 3.10 osx users to use pure python. If performance is important to you, we highly recommend you upgrade to 4.21.7.

[pietrodn]

I confirm the issue is fixed for 3.19.5, 3.19.6, 3.20.2, and 3.20.3 on Python 3.10.6, macOS 12.6, M1 Pro processor. 😄

Thank you!

[nmariz]

I confirm the issue is fixed for 3.19.5, 3.19.6, 3.20.2, and 3.20.3 on Python 3.10.6, macOS 12.6, M1 Pro processor. 😄

No it's not.

$ python --version
Python 3.10.3

$ python -c "import google.protobuf; print(google.protobuf.__version__)"
3.20.3

$ python -c "import google.protobuf.descriptor"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/nmariz/tmp/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: dlopen(/Users/nmariz/tmp/venv/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

And upgrading to 4.21.7 is not a workable solution, since the stubs are incompatible with 3.x. As well as the environment variable, specially for libraries that use Protobuf. I don't get it why the bug is closed and not fixed. Basically all macOS users will have to use a release with a known security issue (in the 3.20 series).

[pietrodn]

@nmariz, somehow the import works perfectly on my Mac with Python 3.10.6 and Protobuf 3.20.3. Very weird. 🤔

[deannagarcia]

Sorry you're still seeing this issue, I thought it was resolved.

I was able to reproduce with the steps in the original bug and after my change the error went away. When you use those steps are you still seeing the error?

Can you provide reproduction steps that include installing protobuf? Have you uninstalled/reinstalled?

[nmariz]

Thanks for considering my comment, but please ignore it. Indeed it doesn't raise the error now, because I was using a binary version with the C extension, that I've installed with pip. I think you've yanked that package from PyPI and it's not available anymore, so for macOS ARM64 only the pure Python is available.

[marinagalenko]

Hello! Not sure if I should comment here since the issue is closed, but I'm experiencing the same issue.

sw_vers
ProductName:        macOS
ProductVersion:     13.0.1
BuildVersion:       22A400

python --version
Python 3.10.7

python -c "import google.protobuf; print(google.protobuf.__version__)"
3.20.3

python -c "import google.protobuf.descriptor"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/marina.galenko/Documents/GSL/hudl-videospa/tests/ui/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in <module>
    from google.protobuf.pyext import _message
ImportError: dlopen(/Users/marina.galenko/Documents/GSL/hudl-videospa/tests/ui/venv/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace '__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_'

[fabiogra]

I solved downgrading Python to 3.9.16 (Apple M1 macOS 12.6)

Python 3.9.16 (main, Mar  8 2023, 04:29:24) 
[Clang 14.0.6 ] :: Anaconda, Inc. on darwin

>>> import google.protobuf; print(google.protobuf.__version__)
3.20.3
>>> import google.protobuf.descriptor
>>>