Some builds of lxml fails to load on Apple M2 (ARM) processors

[ddriddle]

On my MacBook Pro with an Apple M2 Pro processor running on Sonoma 14.4.1, I have trouble loading lxml which causes awscli-login to fail. This appears to be caused by incompatible builds that are being pulled from pypi. This issues does not effect the system python (3.9.6) installed by xcode but does effect python versions installed by pyenv:

$ python -m venv venv
$ python -V
Python 3.8.16
$ pip install awscli-login
$ aws configure set plugins.login awscli_login
$ $ aws login
Traceback (most recent call last):
  File "/Users/ddriddle/.pyenv/versions/3.8.16/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/Users/ddriddle/.pyenv/versions/3.8.16/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli/clidriver.py", line 69, in main
    driver = create_clidriver()
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli/clidriver.py", line 78, in create_clidriver
    load_plugins(session.full_config.get('plugins', {}),
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli/plugin.py", line 58, in _import_plugins
    plugins.append(__import__(path))
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli_login/__init__.py", line 10, in <module>
    from .__main__ import main, logout
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli_login/__main__.py", line 32, in <module>
    from .saml import (
  File "/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/awscli_login/saml.py", line 4, in <module>
    import lxml.etree as ET
ImportError: dlopen(/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/lxml/etree.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'

The issue is caused by importing lxml and can be simply reproduced outside awscli-login like so:

$ python
Python 3.8.16 (default, Apr 16 2024, 12:11:39) 
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lxml.etree as ET
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/lxml/etree.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'
import lxml.etree as ET

A work around is to not use the wheel but instead compile the code from source:

$ python -c 'import lxml.etree as ET'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/lxml/etree.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'
$ pip uninstall lxml
$ pip install git+https://github.com/lxml/lxml.git@lxml-5.2.1
$ python -c 'import lxml.etree as ET'

[ddriddle]

This issue is now effecting the Github macOS runners :-(

======================================================================
ERROR: tests.test_saml (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_saml
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/Users/runner/work/awscli-login/awscli-login/src/tests/test_saml.py", line 12, in <module>
    from lxml.etree import XMLSyntaxError
ImportError: dlopen(/Users/runner/work/awscli-login/awscli-login/.tox/wheel/lib/python3.8/site-packages/lxml/etree.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'

https://github.com/techservicesillinois/awscli-login/actions/runs/8849484810/job/24301956832

It is effecting version 3.8 but not 3.9, 3.10, 3.11 or 3.12 on macOS.

[ddriddle]

Another workaround that works nicely:

$ python -c 'import lxml.etree as ET'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: dlopen(/Users/ddriddle/.pyenv/versions/3.8.16/lib/python3.8/site-packages/lxml/etree.cpython-38-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'
$ pip uninstall lxml
$ PIP_NO_BINARY=lxml pip install lxml
$ python -c 'import lxml.etree as ET'