search

MrMino / wheelfile

🔪🧀 API for creating and inspecting Python .whl files (wheels)
MIT License
29 stars 5 forks source link

Failing to load present METADATA #13

Open mboisson opened 3 years ago

mboisson commented 3 years ago

I am writing a script using wheelfile, and it fails on some wheels that seem otherwise correct. It fails with : 

Traceback (most recent call last):
  File "../manipulate_wheels.py", line 115, in <module>
    main()
  File "../manipulate_wheels.py", line 57, in main
    with WheelFile(w) as wf:
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1224, in __init__
    self.validate()
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1730, in validate
    raise ValueError(
ValueError: METADATA file is not present in the archive or is corrupted.

yet, if I unzip the wheel, there is: 

jupyterlab-3.1.7.dist-info/METADATA

I am not sure what causes this.

mboisson commented 3 years ago

Ah, the issue is that wheelfile does not handle the License-File attribute.

That is in specific. In general, I think wheelfile barfs whenever it encounters an attribute it does not know about. Is this something expected ?

mboisson commented 3 years ago

Ah, I see that this is using https://www.python.org/dev/peps/pep-0639/ which is still in draft.... but I find it is being used in many of the wheels we host already.

MrMino commented 3 years ago

Yea, I've just debugged it and was wondering when did PyPA specify the License-File field. The sad thing is that the wheel in question still uses Metadata-Version: 2.1...

Thanks for pointing this out. I'll probably add it provisionally into the next version even though it's not in the spec yet. Do you happen to have any other examples wheels that use this, just so I can make sure there aren't any other surprises?

mboisson commented 3 years ago

I will run our script https://github.com/ComputeCanada/wheels_builder/blob/main/manipulate_wheels.py on a large sample of our ~6k wheels, see if anything else pops up.

I have patched our local installation of wheelfile.py to include the patch in PR #14, so it won't crash for License-File. I'll see if it crashes on something else.

MrMino commented 3 years ago

From the discussion it seems that the current text is not up to date. I'll have to either wait for the PEP to get updated or read through the messages to get the feel for the consesnus, both of which will take time.

Thanks for the PR. Next version will come after I'll get back from vacation (at the beginning of September).

mboisson commented 3 years ago

Ok, I added some exception handling to wheelfile.py to be able to print a description of the error, and here is what I got: 

$ ./manipulate_wheels.py --wheels /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/*{cp37,py3,any}*.whl --print_req | grep -B1 Exception
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/blist-1.3.6+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/Bottleneck-1.3.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.3.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/gvar-11.9.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/immutables-0.15+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/jaxlib-0.1.64+computecanada-cp37-none-linux_x86_64.whl:
Exception:Invalid version: '0.1.64+computecanada+cuda110'
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/louvain-0.7.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.1.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phono3py-2.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.11.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.9.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.15.4+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.16.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/scikit_image-0.18.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/spglib-1.16.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.3+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.

It looks like Require and Provide are somewhat commonly used.

The invalid version one is obviously our mess up when first patching our wheels without wheelfile, using our crude sed in a shell script.

MrMino commented 3 years ago

Alright, thanks! I'll work it out in the next version.

ccoulombe commented 3 years ago

How is this coming along?

MrMino commented 3 years ago

Not swiftly unfortunately, due to my private life. This is still on my mind though, don't worry.