search

jordansissel / fpm

Effing package management! Build packages for multiple platforms (deb, rpm, etc) with great ease and sanity.
http://fpm.readthedocs.io/en/latest/
Other
11.13k stars 1.07k forks source link

Support Wheel python binary format #587

Open thedrow opened 10 years ago

thedrow commented 10 years ago

See here for details about wheel.

Basically the feature should allow you to download a package from PyPi and turn it into a wheel.

oubiwann commented 10 years ago

I gotta second this. Wheel is rapidly becoming a standard in Python and is a great improvement over eggs. Wheel has been PEP'ed and it's official. Let's do this thing!

atorkhov commented 10 years ago

Why fpm does need to do this? There exist pip wheel tool that does exactly what is requested - "download a package from PyPi and turn it into a wheel".

thedrow commented 10 years ago

Because fpm is the entry point of every artifact building process. Say you want to turn a package to a wheel and then package it as a deb/rpm with upstard/systemd configuration. It's easier to do with one tool. Especially if you use fpm-cookery.

jcockhren commented 10 years ago

:+1:

mcallaway commented 9 years ago

+1

itsgrimetime commented 6 years ago

I'm interested in implementing this, as I'm currently unable to repackage mxnet, mxnet-cu80, or tensorflow-gpu from their pip packages on pypi:

https://github.com/tensorflow/tensorflow/issues/6540

Based on the description here: https://pypi.python.org/pypi/wheel it seems that the .whl format is (in most cases) simply a zip file. Would an alright approach be unzipping this wheel (or installing it via pip install) and building any other packaging type as if the source was a zip?

jordansissel commented 6 years ago

Your proposal sounds like a good start!

On Fri, Oct 20, 2017 at 12:10 PM Mike Grimes notifications@github.com wrote:

I'm interested in implementing this, as I'm currently unable to repackage mxnet, mxnet-cu80, or tensorflow-gpu from their pip packages on pypi:

tensorflow/tensorflow#6540 https://github.com/tensorflow/tensorflow/issues/6540

Based on the description here: https://pypi.python.org/pypi/wheel it seems that the .whl format is (in most cases) simply a zip file. Would an alright approach be unzipping this wheel (or installing it via pip install) and building any other packaging type as if the source was a zip?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jordansissel/fpm/issues/587#issuecomment-338297337, or mute the thread https://github.com/notifications/unsubscribe-auth/AAIC6tuZDDoQtdaU-XG3L-aT685wSr64ks5suPA5gaJpZM4BOS9W .

iFreilicht commented 6 years ago

I've looked into this a little bit as well, I also want to repackage mxnet-cu80, which doesn't have a source package on pypi.

Would an alright approach be unzipping this wheel (or installing it via pip install) and building any other packaging type as if the source was a zip?

Almost. You also need to generate a new setup.py file, as that is not distributed with a wheel. To do this properly, you will need to parse the data contained in the .dist-info and .data directories inside the wheel. A source package had different files for that. Apart from this, there seems to be no difference between a source package and a wheel, as can be seen in PEP427.

To parse the metadata properly, you'll need to look into PEP426-2.0 and PEP426-3.0(The title is incorrect here!). The *.dist-info/metadata.json contains most information needed, but for some special functionality like installable scripts, you will need to look at the *.data directory as well, which is defined in PEP427, linked above.

I've written a python script that can generate a minimal setup.py file. That could be used as a starting point. If you want, I can discuss with my employer whether I can release it.

Maybe it would also be possible to take a look at the implementation of setuptools wheel building process and just reverse that?

etano commented 6 years ago

Hi @iFreilicht , I'd be interested in this script. Trying to distribute as rpm's and tensorflow (for example) is only distributed as a wheel.

neeasade commented 5 years ago

This may be of interest here: https://github.com/montag451/pypi-mirror

I was able to use the output of that script to get setup.py files in tarballs that I could then use FPM to generate packages with

piexil commented 2 years ago

I have run into a few PyPi dependencies that a python module I was attempting to package with fpm was unable to package as source packages where no longer being posted to PyPi, only wheels. I have used this tool as a workaround, it may be of interest. https://github.com/upciti/wheel2deb

I had to do some finesseing with sed on the version strings to make the packages match the package dependencies fpm creates, but it works.