`ImportError: symbol not found` when running `ape test` on macOS

[merc1er]

Environment information

• OS: macOS
• Python Version: 3.10.6
• ape and plugin versions:

$ ape --version
0.5.1

$ ape plugins list
Installed Plugins:
  optimism    0.5.0a1
  vyper       0.5.0

• Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

$ cat ape-config.yaml
plugins:
  - name: vyper
  - name: optimism

What went wrong?

When running ape test (or pytest), I am getting the following error:

ImportError: dlopen(/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_)

Full traceback:

```shell ❯ ape test Traceback (most recent call last): File "/Users/merc1er/.pyenv/versions/ape/bin/ape", line 8, in sys.exit(cli()) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ape/_cli.py", line 40, in invoke return super().invoke(ctx) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func return ctx.invoke(f, obj, *args, **kwargs) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ape_test/_cli.py", line 17, in cli return_code = pytest.main([*pytest_args], ["ape_test"]) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 145, in main config = _prepareconfig(args, plugins) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig config = pluginmanager.hook.pytest_cmdline_parse( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__ return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec return self._inner_hookexec(hook_name, methods, kwargs, firstresult) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_callers.py", line 55, in _multicall gen.send(outcome) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse config: Config = outcome.get_result() File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result raise ex[1].with_traceback(ex[2]) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall res = hook_impl.function(*args) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1017, in pytest_cmdline_parse self.parse(args) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1305, in parse self._preparse(args, addopts=addopts) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/config/__init__.py", line 1188, in _preparse self.pluginmanager.load_setuptools_entrypoints("pytest11") File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints plugin = ep.load() File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/importlib_metadata/__init__.py", line 203, in load module = import_module(match.group('module')) File "/Users/merc1er/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 992, in _find_and_load_unlocked File "", line 241, in _call_with_frames_removed File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/web3/tools/__init__.py", line 1, in from .pytest_ethereum import ( # noqa: F401 File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/web3/tools/pytest_ethereum/deployer.py", line 12, in from ethpm import ( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/__init__.py", line 19, in from .package import Package # noqa: E402, F401 File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/package.py", line 47, in from ethpm.dependencies import ( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/dependencies.py", line 7, in from ethpm.validation.package import ( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/validation/package.py", line 11, in from ethpm._utils.ipfs import ( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module exec(co, module.__dict__) File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/ethpm/_utils/ipfs.py", line 18, in from google.protobuf.descriptor import ( File "/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 47, in from google.protobuf.pyext import _message ImportError: dlopen(/Users/merc1er/.pyenv/versions/ape/lib/python3.10/site-packages/google/protobuf/pyext/_message.cpython-310-darwin.so, 0x0002): symbol not found in flat namespace (__ZN6google8protobuf15FieldDescriptor12TypeOnceInitEPKS1_) ```

This however works fine in GitHub actions (ubuntu-latest).

[merc1er]

How to reproduce:

On macOS (M1 chip), in a clean virtualenv:

pip install eth-ape
ape init
ape test

[fubuloubu]

That's a very strange one, @helloibis has an M1 mac and can maybe help debug this

[ghost]

@merc1er @fubuloubu yep, able to reproduce this! Going to take a look now

[ghost]

@fubuloubu @merc1er seems to be an issue with the web3.py dependency of protobuf, even on the latest web3.py pre-release 6.0.0b5

In the meantime I was able to manually fix the import error by downgrading protobuf by a point version:

(venv) $ pip uninstall protobuf
(venv) $ pip install protobuf==3.20.1

There's an upstream issue at https://github.com/ethereum/web3.py/issues/2654, but I'm not sure how much work getting them on 4.x will be. I run into this issue if I install the latest protobuf rather than downgrading to 3.20.1:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

[antazoey]

3.20.2 was released on Sept. 13th. We could pin to 3.20.1 in Ape. Currently, our web3.py version is pinned anyway, so it wouldn't cause any unexpected problems. Then when we upgrade web3.py to its production release, we can delete the protobuf pin then.

[ghost]

This has been fixed upstream and should be in a release next week

https://github.com/ethereum/web3.py/pull/2657

Once their release is out we can update our dependency!

[merc1er]

Thanks for the quick turnaround.

I can confirm #1071 works with protobuf==3.20.1 and fixes this for me.

[ghost]

Web3 just released their new beta version, so I've opened up https://github.com/ApeWorX/ape/pull/1082 to see about just updating our dependency. Will close out #1071 once we find out if the new PR suffices 🙂