search

kimgr / asn1ate

A Python library for translating ASN.1 into other forms.
Other
69 stars 41 forks source link

asn1ate needs wheels #73

Closed JanneKiiskila closed 1 year ago

JanneKiiskila commented 3 years ago

Related to a finding from Pelion manifest-tool - turns out the issue is actually in asn1ate.

https://github.com/PelionIoT/manifest-tool/issues/27

Short conclusion: asn1ate needs wheels, it should ask for it in it's requirements.txt.

kimgr commented 3 years ago

@JanneKiiskila Thanks for the report. I wonder why this requirement comes about though. asn1ate itself has no dependency on wheels as far as I can see. The buildlog you quoted says:

...
Building wheels for collected packages: asn1ate, pyrsistent
  Building wheel for asn1ate (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/jankii01/pelion-venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-q4ls89jh/asn1ate/setup.py'"'"'; __file__='"'"'/tmp/pip-install-q4ls89jh/asn1ate/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-_crew3o_
       cwd: /tmp/pip-install-q4ls89jh/asn1ate/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: invalid command 'bdist_wheel'
  ----------------------------------------
...

so I wonder why pip for the outer package tries to invoke setup.py bdist_wheel...?

JanneKiiskila commented 3 years ago

This is from one of my colleagues @Moshe Shahar

After investigating a bit more, this behavior has changed from pip v20.1.1, and from that version, you will see: Using legacy 'setup.py install' for pyrsistent, since package 'wheel' is not installed.

The correct fix is to add setup_requires = wheel to setup.py in the packages that don’t provide pre-built wheels such as pyrsistent.

JanneKiiskila commented 3 years ago

This is a strange issue, seems we're not the only ones with this type of issues: https://stackoverflow.com/questions/34819221/why-is-python-setup-py-saying-invalid-command-bdist-wheel-on-travis-ci

kimgr commented 3 years ago

@JanneKiiskila Thanks. So it sounds to me this is a bug in pip? I don't want to add setup_requires to asn1ate when it doesn't in fact require wheel.

Think of it from the viewpoint of a freestanding consumer -- pip install asn1ate does not require wheel, so why should it be installed in order to install asn1ate?

Would it help if I published an asn1ate wheel to pypi?

JanneKiiskila commented 3 years ago

Based on @moshe-shahar 's comment that might help. If you do that, I can do a test on it, either with a clean new virtual machine or a Docker image, I believe that should also reproduce the issue we're seeing.

moshe-shahar commented 3 years ago

@kimgr , yes, publishing of a wheel should fix that. when testing, I suggest using --no-cache-dir: pip install --no-cache-dir -v asn1ate

kimgr commented 3 years ago

@moshe-shahar @JanneKiiskila Thanks for the testing tip, that came in handy. I've published wheels for 0.6.0 to pypi now, and they appear to be installable:

# python3
$ cd /tmp/
$ virtualenv testwhl && . testwhl/bin/activate
created virtual environment CPython3.8.5.final.0-64 in 189ms
  creator CPython3Posix(dest=/tmp/testwhl, clear=False, global=False)
  seeder FromAppData(download=False, distlib=latest, pyparsing=latest, html5lib=latest, appdirs=latest, progress=latest, pytoml=latest, requests=latest, ipaddr=latest, pep517=latest, certifi=latest, lockfile=latest, webencodings=latest, six=latest, distro=latest, CacheControl=latest, pip=latest, retrying=latest, chardet=latest, colorama=latest, msgpack=latest, pkg_resources=latest, wheel=latest, idna=latest, packaging=latest, setuptools=latest, urllib3=latest, contextlib2=latest, via=copy, app_data_dir=...)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
(testwhl) $ pip install --no-cache-dir -v asn1ate
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-1by0ve18
Created temporary directory: /tmp/pip-req-tracker-kh4k1dk5
Initialized build tracking at /tmp/pip-req-tracker-kh4k1dk5
Created build tracker: /tmp/pip-req-tracker-kh4k1dk5
Entered build tracker: /tmp/pip-req-tracker-kh4k1dk5
Created temporary directory: /tmp/pip-install-kl4rkyes
1 location(s) to search for versions of asn1ate:
* https://pypi.org/simple/asn1ate/
Fetching project page and analyzing links: https://pypi.org/simple/asn1ate/
Getting page https://pypi.org/simple/asn1ate/
Found index url https://pypi.org/simple
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/asn1ate/ HTTP/1.1" 200 888
  Found link https://files.pythonhosted.org/packages/07/ab/1bd5c0ede3899d526edc6bee1f6dd83f30a15edc012122a9aaabb625366a/asn1ate-0.4.tar.gz#sha256=1b9979c05e6c71d10686fc3327dd102c70562a872b8f27b208850dacef5e816f (from https://pypi.org/simple/asn1ate/), version: 0.4
  Found link https://files.pythonhosted.org/packages/dc/40/77a708eb914d32be6329d66afb20a8715de75b5c28c56b273d116185d4ed/asn1ate-0.4.zip#sha256=2af5e827e2e98b10330be06f8d821584867abeec601017c92198be442476418f (from https://pypi.org/simple/asn1ate/), version: 0.4
  Found link https://files.pythonhosted.org/packages/2d/1f/0105593d84305e80bd9281eede3498c137d49b62776c84adf4b1e6ea47f4/asn1ate-0.5.tar.gz#sha256=f3c84b3a626e2251e8f64d5e594512c8939584872491a3f82d04bc52dc416652 (from https://pypi.org/simple/asn1ate/), version: 0.5
  Found link https://files.pythonhosted.org/packages/65/59/e3fdcbb7dc2e19ab5b132901a4115446070b20bccac86945f3e78ce5f32d/asn1ate-0.5.zip#sha256=e789432a3b085a281c4329df51f51d7d66834f4d878db45ca548fd700abe9278 (from https://pypi.org/simple/asn1ate/), version: 0.5
  Skipping link: none of the wheel's tags match: py2-none-any: https://files.pythonhosted.org/packages/68/7e/00dd31317ecfbc67580d14de5c3af9c36978b8b46224cdc248e088a097db/asn1ate-0.6.0-py2-none-any.whl#sha256=8f0ded3f26e3d5d7cc1d40b7b9d374c13992842fa2bb0885c10fb462d01f6293 (from https://pypi.org/simple/asn1ate/)
  Found link https://files.pythonhosted.org/packages/c6/1f/96f09ecca1ebc436327fb6c8148f9a24923d5e735689f86262b03e0af12c/asn1ate-0.6.0-py3-none-any.whl#sha256=521166d237e7d559913317111b31cadee92619282e152228417c195d6d9d3a9f (from https://pypi.org/simple/asn1ate/), version: 0.6.0
  Found link https://files.pythonhosted.org/packages/ec/20/7310475955a893ddd0beb7c8a9b1f940390d67bc46cae9d60c7bc7566d5c/asn1ate-0.6.0.tar.gz#sha256=fe2cb928b1187cbd255272f2dc1df600f42d8810845b65e0f8d0551fbcd2c7ab (from https://pypi.org/simple/asn1ate/), version: 0.6.0
Given no hashes to check 6 links for project 'asn1ate': discarding no candidates
Using version 0.6.0 (newest of versions: 0.4, 0.5, 0.6.0)
Collecting asn1ate
  Created temporary directory: /tmp/pip-unpack-2rwutn2b
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/c6/1f/96f09ecca1ebc436327fb6c8148f9a24923d5e735689f86262b03e0af12c/asn1ate-0.6.0-py3-none-any.whl HTTP/1.1" 200 29225
  Downloading asn1ate-0.6.0-py3-none-any.whl (29 kB)
  Added asn1ate from https://files.pythonhosted.org/packages/c6/1f/96f09ecca1ebc436327fb6c8148f9a24923d5e735689f86262b03e0af12c/asn1ate-0.6.0-py3-none-any.whl#sha256=521166d237e7d559913317111b31cadee92619282e152228417c195d6d9d3a9f to build tracker '/tmp/pip-req-tracker-kh4k1dk5'
  Removed asn1ate from https://files.pythonhosted.org/packages/c6/1f/96f09ecca1ebc436327fb6c8148f9a24923d5e735689f86262b03e0af12c/asn1ate-0.6.0-py3-none-any.whl#sha256=521166d237e7d559913317111b31cadee92619282e152228417c195d6d9d3a9f from build tracker '/tmp/pip-req-tracker-kh4k1dk5'
Requirement already satisfied: pyparsing>=2.0.0 in ./testwhl/lib/python3.8/site-packages (from asn1ate) (2.4.6)
Installing collected packages: asn1ate
  Created temporary directory: /tmp/pip-unpacked-wheel-af_6a8d2

  changing mode of /tmp/testwhl/bin/asn1ate to 775
Successfully installed asn1ate-0.6.0
Cleaning up...
Removed build tracker: '/tmp/pip-req-tracker-kh4k1dk5'
(testwhl) $ deactivate && rm -rf testwhl

and:

# python2
$ virtualenv -p python2 testwhl && . testwhl/bin/activate
created virtual environment CPython2.7.18.final.0-64 in 251ms
  creator CPython2Posix(dest=/tmp/testwhl, clear=False, global=False)
  seeder FromAppData(download=False, distlib=latest, pyparsing=latest, html5lib=latest, appdirs=latest, progress=latest, pytoml=latest, requests=latest, ipaddr=latest, pep517=latest, certifi=latest, lockfile=latest, webencodings=latest, six=latest, distro=latest, CacheControl=latest, pip=latest, retrying=latest, chardet=latest, colorama=latest, msgpack=latest, pkg_resources=latest, wheel=latest, idna=latest, packaging=latest, setuptools=latest, urllib3=latest, contextlib2=latest, via=copy, app_data_dir=...)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
(testwhl) $ pip install --no-cache-dir -v asn1ate
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-EvdEJe
Created temporary directory: /tmp/pip-req-tracker-tpqFRs
Initialized build tracking at /tmp/pip-req-tracker-tpqFRs
Created build tracker: /tmp/pip-req-tracker-tpqFRs
Entered build tracker: /tmp/pip-req-tracker-tpqFRs
Created temporary directory: /tmp/pip-install-4dUM2H
1 location(s) to search for versions of asn1ate:
* https://pypi.org/simple/asn1ate/
Fetching project page and analyzing links: https://pypi.org/simple/asn1ate/
Getting page https://pypi.org/simple/asn1ate/
Found index url https://pypi.org/simple
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/asn1ate/ HTTP/1.1" 200 888
  Found link https://files.pythonhosted.org/packages/07/ab/1bd5c0ede3899d526edc6bee1f6dd83f30a15edc012122a9aaabb625366a/asn1ate-0.4.tar.gz#sha256=1b9979c05e6c71d10686fc3327dd102c70562a872b8f27b208850dacef5e816f (from https://pypi.org/simple/asn1ate/), version: 0.4
  Found link https://files.pythonhosted.org/packages/dc/40/77a708eb914d32be6329d66afb20a8715de75b5c28c56b273d116185d4ed/asn1ate-0.4.zip#sha256=2af5e827e2e98b10330be06f8d821584867abeec601017c92198be442476418f (from https://pypi.org/simple/asn1ate/), version: 0.4
  Found link https://files.pythonhosted.org/packages/2d/1f/0105593d84305e80bd9281eede3498c137d49b62776c84adf4b1e6ea47f4/asn1ate-0.5.tar.gz#sha256=f3c84b3a626e2251e8f64d5e594512c8939584872491a3f82d04bc52dc416652 (from https://pypi.org/simple/asn1ate/), version: 0.5
  Found link https://files.pythonhosted.org/packages/65/59/e3fdcbb7dc2e19ab5b132901a4115446070b20bccac86945f3e78ce5f32d/asn1ate-0.5.zip#sha256=e789432a3b085a281c4329df51f51d7d66834f4d878db45ca548fd700abe9278 (from https://pypi.org/simple/asn1ate/), version: 0.5
  Found link https://files.pythonhosted.org/packages/68/7e/00dd31317ecfbc67580d14de5c3af9c36978b8b46224cdc248e088a097db/asn1ate-0.6.0-py2-none-any.whl#sha256=8f0ded3f26e3d5d7cc1d40b7b9d374c13992842fa2bb0885c10fb462d01f6293 (from https://pypi.org/simple/asn1ate/), version: 0.6.0
  Skipping link: none of the wheel's tags match: py3-none-any: https://files.pythonhosted.org/packages/c6/1f/96f09ecca1ebc436327fb6c8148f9a24923d5e735689f86262b03e0af12c/asn1ate-0.6.0-py3-none-any.whl#sha256=521166d237e7d559913317111b31cadee92619282e152228417c195d6d9d3a9f (from https://pypi.org/simple/asn1ate/)
  Found link https://files.pythonhosted.org/packages/ec/20/7310475955a893ddd0beb7c8a9b1f940390d67bc46cae9d60c7bc7566d5c/asn1ate-0.6.0.tar.gz#sha256=fe2cb928b1187cbd255272f2dc1df600f42d8810845b65e0f8d0551fbcd2c7ab (from https://pypi.org/simple/asn1ate/), version: 0.6.0
Given no hashes to check 6 links for project 'asn1ate': discarding no candidates
Using version 0.6.0 (newest of versions: 0.4, 0.5, 0.6.0)
Collecting asn1ate
  Created temporary directory: /tmp/pip-unpack-zWROV3
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/68/7e/00dd31317ecfbc67580d14de5c3af9c36978b8b46224cdc248e088a097db/asn1ate-0.6.0-py2-none-any.whl HTTP/1.1" 200 29225
  Downloading asn1ate-0.6.0-py2-none-any.whl (29 kB)
  Added asn1ate from https://files.pythonhosted.org/packages/68/7e/00dd31317ecfbc67580d14de5c3af9c36978b8b46224cdc248e088a097db/asn1ate-0.6.0-py2-none-any.whl#sha256=8f0ded3f26e3d5d7cc1d40b7b9d374c13992842fa2bb0885c10fb462d01f6293 to build tracker '/tmp/pip-req-tracker-tpqFRs'
  Removed asn1ate from https://files.pythonhosted.org/packages/68/7e/00dd31317ecfbc67580d14de5c3af9c36978b8b46224cdc248e088a097db/asn1ate-0.6.0-py2-none-any.whl#sha256=8f0ded3f26e3d5d7cc1d40b7b9d374c13992842fa2bb0885c10fb462d01f6293 from build tracker '/tmp/pip-req-tracker-tpqFRs'
Requirement already satisfied: pyparsing>=2.0.0 in ./testwhl/lib/python2.7/site-packages (from asn1ate) (2.4.6)
Installing collected packages: asn1ate
  Created temporary directory: /tmp/pip-unpacked-wheel-cPqCyX

  changing mode of /tmp/testwhl/bin/asn1ate to 775
Successfully installed asn1ate-0.6.0
Cleaning up...
Removed build tracker: '/tmp/pip-req-tracker-tpqFRs'
(testwhl) $ deactivate && rm -rf testwhl

Hope that helps!

JanneKiiskila commented 1 year ago

Thank you, we've not had any issues since so I'd say this bug is fixed.

kimgr commented 1 year ago

Thanks for the extensive test period, I had no memory of having done this 🙂