search

piwheels / packages

Issue tracker for piwheels package issues
https://github.com/piwheels/packages/issues
20 stars 5 forks source link

aioxmpp installs the wrong dependencies for Python 3.5+ #172

Closed slovdahl closed 3 years ago

slovdahl commented 3 years ago

Package name: aioxmpp Link to PyPI page: https://pypi.org/project/aioxmpp/ Link to piwheels page: https://www.piwheels.org/project/aioxmpp/ Version: 0.11.0 at least Python version: 3.7 at least I am the maintainer: No More information: Originated in https://github.com/horazont/aioxmpp/issues/350

Installing the aioxmpp 0.11.0 with Python 3.7 and piwheels enabled causes typing to be installed and constrains lxml in the wrong way. According to https://github.com/horazont/aioxmpp/blob/6806113c48839dab55460679f9f8629c3fe39346/setup.py#L59 and https://github.com/horazont/aioxmpp/blob/6806113c48839dab55460679f9f8629c3fe39346/setup.py#L38, that should only be done if Python version <= 3.4.

The reason this is a problem is that:

  1. Home Assistant nowadays blocks typing from being installed.
  2. I use a component in HASS called bosch-thermostat-client that depends on aioxmpp.
  3. Installing Home Assistant in a new venv fails because of the two points above.

With piwheels enabled:

slovdahl@rpi:~/test-venv $ python3.7 -m venv venv-3.7
slovdahl@rpi:~/test-venv $ cd venv-3.7/
slovdahl@rpi:~/test-venv/venv-3.7 $ source bin/activate
(venv-3.7) slovdahl@rpi:~/test-venv/venv-3.7 $ pip3 install aioxmpp
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting aioxmpp
  Downloading https://www.piwheels.org/simple/aioxmpp/aioxmpp-0.11.0-py3-none-any.whl (469kB)
    100% |████████████████████████████████| 471kB 622kB/s
Collecting lxml<4.4,~=4.0 (from aioxmpp)
  Downloading https://www.piwheels.org/simple/lxml/lxml-4.3.5-cp37-cp37m-linux_armv7l.whl (6.0MB)
    100% |████████████████████████████████| 6.0MB 76kB/s
Collecting aioopenssl>=0.1 (from aioxmpp)
  Downloading https://www.piwheels.org/simple/aioopenssl/aioopenssl-0.5.1-py3-none-any.whl
Collecting pyOpenSSL (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl
Collecting babel~=2.3 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/dd/a5/81076e10b5ef74493cf08a8e419e61b64324c9c55db4aa7f89c0240c4873/Babel-2.9.0-py2.py3-none-any.whl
Collecting pyasn1 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl
Collecting sortedcollections>=0.5 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ea/f1/831fa9763afc85d3065557762824490d6e4d1afe1a56e0f053a568ae0cb5/sortedcollections-1.2.1-py2.py3-none-any.whl
Collecting multidict<5,>=2.0 (from aioxmpp)
  Using cached https://www.piwheels.org/simple/multidict/multidict-4.7.6-cp37-cp37m-linux_armv7l.whl
Collecting aiosasl>=0.3 (from aioxmpp)
  Downloading https://www.piwheels.org/simple/aiosasl/aiosasl-0.4.1-py3-none-any.whl
Collecting tzlocal~=1.2 (from aioxmpp)
  Downloading https://www.piwheels.org/simple/tzlocal/tzlocal-1.5.1-py3-none-any.whl
Collecting pyasn1-modules (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl
Collecting dnspython~=1.0 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl
Collecting typing (from aioxmpp)
  Downloading https://www.piwheels.org/simple/typing/typing-3.7.4.3-py3-none-any.whl
Collecting six>=1.5.2 (from pyOpenSSL->aioxmpp)
  Downloading https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting cryptography>=2.8 (from pyOpenSSL->aioxmpp)
  Using cached https://www.piwheels.org/simple/cryptography/cryptography-3.2.1-cp37-cp37m-linux_armv7l.whl
Collecting pytz>=2015.7 (from babel~=2.3->aioxmpp)
  Downloading https://files.pythonhosted.org/packages/12/f8/ff09af6ff61a3efaad5f61ba5facdf17e7722c4393f7d8a66674d2dbd29f/pytz-2020.4-py2.py3-none-any.whl (509kB)
    100% |████████████████████████████████| 512kB 841kB/s
Collecting sortedcontainers (from sortedcollections>=0.5->aioxmpp)
  Using cached https://files.pythonhosted.org/packages/20/4d/a7046ae1a1a4cc4e9bbed194c387086f06b25038be596543d026946330c9/sortedcontainers-2.3.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.8->pyOpenSSL->aioxmpp)
  Using cached https://www.piwheels.org/simple/cffi/cffi-1.14.3-cp37-cp37m-linux_armv7l.whl
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.8->pyOpenSSL->aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Installing collected packages: lxml, six, pycparser, cffi, cryptography, pyOpenSSL, aioopenssl, pytz, babel, pyasn1, sortedcontainers, sortedcollections, multidict, aiosasl, tzlocal, pyasn1-modules, dnspython, typing, aioxmpp
Successfully installed aioopenssl-0.5.1 aiosasl-0.4.1 aioxmpp-0.11.0 babel-2.9.0 cffi-1.14.3 cryptography-3.2.1 dnspython-1.16.0 lxml-4.3.5 multidict-4.7.6 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pytz-2020.4 six-1.15.0 sortedcollections-1.2.1 sortedcontainers-2.3.0 typing-3.7.4.3 tzlocal-1.5.1

Without piwheels enabled:

slovdahl@rpi:~/test-venv $ python3.7 -m venv venv-3.7
slovdahl@rpi:~/test-venv $ cd venv-3.7/
slovdahl@rpi:~/test-venv/venv-3.7 $ source bin/activate
(venv-3.7) slovdahl@rpi:~/test-venv/venv-3.7 $ pip3 install aioxmpp
Collecting aioxmpp
  Downloading https://files.pythonhosted.org/packages/dc/0c/987a30bf6d68f9b4ec2bda3ddc0ba7d34c9b0ee5aca91ec3d0f919f18fb2/aioxmpp-0.11.0.tar.gz (378kB)
    100% |████████████████████████████████| 378kB 1.0MB/s
Collecting aioopenssl>=0.1 (from aioxmpp)
  Downloading https://files.pythonhosted.org/packages/b1/34/e28adb7600d8ccff787a841241bc2591a3661a66e4f171d38956c0cd3002/aioopenssl-0.5.1.tar.gz
Collecting aiosasl>=0.3 (from aioxmpp)
  Downloading https://files.pythonhosted.org/packages/c4/5c/6522806fd01028c3b17e2309de6e4d2a989e0bde6f888a6ec70b9e1226b0/aiosasl-0.4.1.tar.gz
Collecting babel~=2.3 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/dd/a5/81076e10b5ef74493cf08a8e419e61b64324c9c55db4aa7f89c0240c4873/Babel-2.9.0-py2.py3-none-any.whl
Collecting dnspython~=1.0 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl
Collecting lxml~=4.0 (from aioxmpp)
  Downloading https://files.pythonhosted.org/packages/c5/2f/a0d8aa3eee6d53d5723d89e1fc32eee11e76801b424e30b55c7aa6302b01/lxml-4.6.1.tar.gz (3.2MB)
    100% |████████████████████████████████| 3.2MB 142kB/s
Collecting multidict<5,>=2.0 (from aioxmpp)
  Downloading https://files.pythonhosted.org/packages/65/d4/fabdcc5ee4451c8a8e177e27ddfd131a53a82ecc5a3b68468b7e9f8d70b4/multidict-4.7.6.tar.gz (50kB)
    100% |████████████████████████████████| 51kB 3.5MB/s
  Installing build dependencies ... done
Collecting pyOpenSSL (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/9e/de/f8342b68fa9e981d348039954657bdf681b2ab93de27443be51865ffa310/pyOpenSSL-19.1.0-py2.py3-none-any.whl
Collecting pyasn1 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/62/1e/a94a8d635fa3ce4cfc7f506003548d0a2447ae76fd5ca53932970fe3053f/pyasn1-0.4.8-py2.py3-none-any.whl
Collecting pyasn1_modules (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/95/de/214830a981892a3e286c3794f41ae67a4495df1108c3da8a9f62159b9a9d/pyasn1_modules-0.2.8-py2.py3-none-any.whl
Collecting sortedcollections>=0.5 (from aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ea/f1/831fa9763afc85d3065557762824490d6e4d1afe1a56e0f053a568ae0cb5/sortedcollections-1.2.1-py2.py3-none-any.whl
Collecting tzlocal~=1.2 (from aioxmpp)
  Downloading https://files.pythonhosted.org/packages/cb/89/e3687d3ed99bc882793f82634e9824e62499fdfdc4b1ae39e211c5b05017/tzlocal-1.5.1.tar.gz
Collecting pytz>=2015.7 (from babel~=2.3->aioxmpp)
  Using cached https://files.pythonhosted.org/packages/12/f8/ff09af6ff61a3efaad5f61ba5facdf17e7722c4393f7d8a66674d2dbd29f/pytz-2020.4-py2.py3-none-any.whl
Collecting cryptography>=2.8 (from pyOpenSSL->aioxmpp)
  Downloading https://files.pythonhosted.org/packages/94/5c/42de91c7fbdb817b2d9a4e64b067946eb38a4eb36c1a09c96c87a0f86a82/cryptography-3.2.1.tar.gz (540kB)
    100% |████████████████████████████████| 542kB 786kB/s
  Installing build dependencies ... done
Collecting six>=1.5.2 (from pyOpenSSL->aioxmpp)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl
Collecting sortedcontainers (from sortedcollections>=0.5->aioxmpp)
  Cache entry deserialization failed, entry ignored
  Using cached https://files.pythonhosted.org/packages/20/4d/a7046ae1a1a4cc4e9bbed194c387086f06b25038be596543d026946330c9/sortedcontainers-2.3.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography>=2.8->pyOpenSSL->aioxmpp)
  Using cached https://files.pythonhosted.org/packages/cb/ae/380e33d621ae301770358eb11a896a34c34f30db188847a561e8e39ee866/cffi-1.14.3.tar.gz
Collecting pycparser (from cffi!=1.11.3,>=1.8->cryptography>=2.8->pyOpenSSL->aioxmpp)
  Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Building wheels for collected packages: aioxmpp, aioopenssl, aiosasl, lxml, multidict, tzlocal, cryptography, cffi
  Running setup.py bdist_wheel for aioxmpp ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/aioxmpp/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-yet0m301 --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for aioxmpp
  Running setup.py clean for aioxmpp
  Running setup.py bdist_wheel for aioopenssl ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/aioopenssl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-z1sad67k --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for aioopenssl
  Running setup.py clean for aioopenssl
  Running setup.py bdist_wheel for aiosasl ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/aiosasl/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-n6z6ye7v --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for aiosasl
  Running setup.py clean for aiosasl
  Running setup.py bdist_wheel for lxml ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/lxml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-yhil_2s_ --python-tag cp37:
  Building lxml version 4.6.1.
  Building without Cython.
  Building against libxml2 2.9.4 and libxslt 1.1.32
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for lxml
  Running setup.py clean for lxml
  Running setup.py bdist_wheel for multidict ... done
  Stored in directory: /home/slovdahl/.cache/pip/wheels/1f/9d/78/5f26d5ea7d1cf9c9df8dbb2b1ecb7a988af034cc1a8cf69a64
  Running setup.py bdist_wheel for tzlocal ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/tzlocal/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-vknnqcl4 --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for tzlocal
  Running setup.py clean for tzlocal
  Running setup.py bdist_wheel for cryptography ... done
  Stored in directory: /home/slovdahl/.cache/pip/wheels/93/82/ae/e8b455fc66cb84eb1e07f29af34d888f37fa62fb707b56f57c
  Running setup.py bdist_wheel for cffi ... error
  Complete output from command /home/slovdahl/test-venv/venv-3.7/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-kix8af0f/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-3ej5djr3 --python-tag cp37:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

  ----------------------------------------
  Failed building wheel for cffi
  Running setup.py clean for cffi
Successfully built multidict cryptography
Failed to build aioxmpp aioopenssl aiosasl lxml tzlocal cffi
Installing collected packages: six, pycparser, cffi, cryptography, pyOpenSSL, aioopenssl, aiosasl, pytz, babel, dnspython, lxml, multidict, pyasn1, pyasn1-modules, sortedcontainers, sortedcollections, tzlocal, aioxmpp
  Running setup.py install for cffi ... done
  Running setup.py install for aioopenssl ... done
  Running setup.py install for aiosasl ... done
  Running setup.py install for lxml ... done
  Running setup.py install for tzlocal ... done
  Running setup.py install for aioxmpp ... done
Successfully installed aioopenssl-0.5.1 aiosasl-0.4.1 aioxmpp-0.11.0 babel-2.9.0 cffi-1.14.3 cryptography-3.2.1 dnspython-1.16.0 lxml-4.6.1 multidict-4.7.6 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pytz-2020.4 six-1.15.0 sortedcollections-1.2.1 sortedcontainers-2.3.0 tzlocal-1.5.1
bennuttall commented 3 years ago

Ok, this happens because that setup.py has runtime conditionals, but generates a wheel which claims to work for all Python 3 versions i.e. aioxmpp-0.11.0-py3-none-any.whl.

The setup.py is executed by piwheels in order to build the wheel, and that conditional is checked at runtime only. The conditional does not make its way into the wheel, as setup.py is not part of the wheel, only the result of running it is. If the outcome of setup.py is that the dependencies include something, the wheel doesn't know what conditions were met in order to create that set of dependencies. It's supposed to be declarative, not runtime-dependent, which is why the Python ecosystem has made an effort to move to a strictly declarative system (i.e. setup.cfg or pyproject.toml).

In piwheels, we attempt builds on the lowest supported ABI first, then move higher until all supported ABIs are covered (either by a pure python wheel, or platform wheels for all). Therefore we attempted on cp34m, succeeded with a pure python wheel, and so stopped there. Unfortunately that had the consequence of being stuck with the cp34m-specific requirements.

Seeing as cp34m is now deprecated, the easiest solution is for us to remove that wheel, let it rebuild on the lowest supported ABI (now cp35m), which should yield a wheel without this legacy dependency.

I'll kick it off now, so you should soon see the file hash change from 965d59748a43f244eabef0abfe0bcdc8c4ec1a87e8fc78775f0a80c339013a40 to something else. Then you know it's a new file that's been built on Stretch / cp35m. Shouldn't take long to make its way through the system - so maybe a few minutes.

bennuttall commented 3 years ago

It's done now - can you confirm this fixes the problem?

slovdahl commented 3 years ago

Thank you very much for the quick response and fix! I can confirm that pip3 install aioxmpp no longer installs typing. 👌