Edit Travis config to build a Python wheel (.whl) along with the .tar.gz file.

darkvertex commented 6 years ago

Hello!

I noticed pyblish-qml was building Python wheels but this core repo and pyblish-lite were not.

As per the Travis docs this tiny change should produce a .whl file as well as the .tar.gz already being made.

Python wheels are the most modern distribution format for pip packages. Pip will use the .whl when it is available and compared to .tar.gz they don't require extracting anything after pip downloads the file, and pip doesn't have to compile the extracted files to pregenerate the pycs either. They're basically a zipped package.

You can read more about them here: https://wheel.readthedocs.io/en/stable/

darkvertex commented 6 years ago

Here are PRs for some of the other repos:

pyblish-lite: https://github.com/pyblish/pyblish-lite/pull/105
pyblish-maya: https://github.com/pyblish/pyblish-maya/pull/74
pyblish-houdini: https://github.com/pyblish/pyblish-houdini/pull/17

mottosso commented 6 years ago

Thanks for this @darkvertex, however a source distribution is actually intentional, as it highlights the fact that there are no compiled portions of this project being distributed. I expect that is useful information to the end user in that (1) it is obviously cross-platform and (2) can safely be stored in a "global" repository of Python libraries (and thereafter shouldn't be compiled .pyc or __pycache__), as well as (3) as part of a larger project that is also cross-platform. A wheel on the other hand says "I'm built for the Python distribution you used to install me on". Great for libraries such as PyQt, less great for pure-Python libraries.

That said, things may have changed since I put that distribution together (~4 years ago) and wheels is one of the things I have less experience with distributing. Have you experienced problems with the current source distribution that prompted or would be solved by a wheel?

darkvertex commented 6 years ago

I feel there's a few misconceptions about source and wheel distributions and how they coexist. Allow me to clarify a few things:

This does not mean your source distribution is not generated; it still will be, but a wheel file will be generated alongside it as well.

I also had some misconceptions of my own and after some deeper reading, I retract part of my statement.. when I said pip was not extracting anything I was wrong, it does have to extract to install it, but with a .whl it extracts once whereas with a .tar.gz it generates a .whl under the hood then extracts that, so extracts twice, in a way. When I said about pip not having to pregenerate pycs, I was wrong, it does it either way. I stand corrected.

Back to your comment though:

(1) it is obviously cross-platform

Indeed, and this reminded me I was missing a setup.cfg to tell bdist_wheel to generate a "universal" wheel by default. Just added it. It now makes 1 wheel file for py2 and py3 together targeting "any" (all) platforms. Woohoo!

and (2) can safely be stored in a "global" repository of Python libraries (and thereafter shouldn't be compiled .pyc or pycache),

Sure, one can extract the .tar.gz of the source distribution into a folder and source that everywhere over the network. Same can be done with a .whl actually. That said, sourcing Python stuff over the network can be slow though, so if you can work with virtualenvs and local pip deployments you will get better performance.

A wheel on the other hand says "I'm built for the Python distribution you used to install me on".

This is the default behaviour but not entirely accurate. First of all, an OS-agnostic build will target the word "any" for the platform, so all OSs are targeted, and by default if you run python setup.py bdist_wheel yes you will get a file called pyblish_base-1.7.0-py2-none-any.whl but if you know your project works for Python 2 and 3, you can make what's known as a "universal" wheel in one of two ways:

Run python setup.py bdist_wheel --universal
Add a setup.cfg file with the lines and then you don't need to pass the --universal flag anymore:
```
[bdist_wheel]
universal = 1
```

Doing this will produce a file called pyblish_base-1.7.0-py2.py3-none-any.whl which, as the name implies, targets not only "any" OS but "py2" and "py3", all in one. Win win! :D :rainbow:

Great for libraries such as PyQt, less great for pure-Python libraries.

It's great for all libraries, compiled or not, period. You're right that for compiled it is a substantial improvement when a wheel exists for your platform as you save on the compilation step. For something big like PyQt, you can save several minutes in your pip install!

That said, things may have changed since I put that distribution together (~4 years ago) and wheels is one of the things I have less experience with distributing.

I don't know if they were 4 years ago, but today wheels are the current standard for pip packages. Anyone not using wheels is on an outdated workflow.

As a matter of fact -- and this is the biggest pro about doing a wheel in my opinion -- did you know pip will build a wheel when only a source distribution is available anyways? By providing a wheel, even a "universal py2+py3 any-platform" one, you save a lot of install processing. Let me show you with pip's verbose (-v) flag...

Here I ran python setup.py bdist_wheel, then proceeded to install the wheel:

> pip install --user -v ~/github/pyblish-base/dist/pyblish_base-1.7.0-py2.py3-none-any.whl
Created temporary directory: /tmp/pip-ephem-wheel-cache-DQCCcu
Created temporary directory: /tmp/pip-req-tracker-UlTa6Y
Created requirements tracker '/tmp/pip-req-tracker-UlTa6Y'
Created temporary directory: /tmp/pip-install-qP8FCm
Processing /home/alan/github/pyblish-base/dist/pyblish_base-1.7.0-py2.py3-none-any.whl
  Added pyblish-base==1.7.0 from file:///home/alan/github/pyblish-base/dist/pyblish_base-1.7.0-py2.py3-none-any.whl to build tracker '/tmp/pip-req-tracker-UlTa6Y'
  Removed pyblish-base==1.7.0 from file:///home/alan/github/pyblish-base/dist/pyblish_base-1.7.0-py2.py3-none-any.whl from build tracker '/tmp/pip-req-tracker-UlTa6Y'
Installing collected packages: pyblish-base

  changing mode of /home/alan/.local/bin/pyblish to 777
Successfully installed pyblish-base-1.7.0
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-UlTa6Y'

Short and sweet. 👍

Now let's look what it looks like installing the source distribution. I went ahead and downloaded the .tar.gz locally for a fair comparison so it's not showing prints for downloading anything from pypi, and of course I did pip remove pyblish_base before running this to uninstall the files first:

> pip install --user -v pyblish-base-1.6.1.tar.gz
Created temporary directory: /tmp/pip-ephem-wheel-cache-HIHVcF
Created temporary directory: /tmp/pip-req-tracker-2VsyFl
Created requirements tracker '/tmp/pip-req-tracker-2VsyFl'
Created temporary directory: /tmp/pip-install-4lFRnB
Processing ./pyblish-base-1.6.1.tar.gz
  Created temporary directory: /tmp/pip-req-build-l17biy
  Added file:///mnt/c/users/alan/Documents/GitHub/pyblish-base-1.6.1.tar.gz to build tracker '/tmp/pip-req-tracker-2VsyFl'
  Running setup.py (path:/tmp/pip-req-build-l17biy/setup.py) egg_info for package from file:///mnt/c/users/alan/Documents/GitHub/pyblish-base-1.6.1.tar.gz
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/pyblish_base.egg-info
    writing pip-egg-info/pyblish_base.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pyblish_base.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pyblish_base.egg-info/dependency_links.txt
    writing entry points to pip-egg-info/pyblish_base.egg-info/entry_points.txt
    writing manifest file 'pip-egg-info/pyblish_base.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/pyblish_base.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/pyblish_base.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-l17biy has version 1.6.1, which satisfies requirement pyblish-base==1.6.1 from file:///mnt/c/users/alan/Documents/GitHub/pyblish-base-1.6.1.tar.gz
  Removed pyblish-base==1.6.1 from file:///mnt/c/users/alan/Documents/GitHub/pyblish-base-1.6.1.tar.gz from build tracker '/tmp/pip-req-tracker-2VsyFl'
Building wheels for collected packages: pyblish-base
  Created temporary directory: /tmp/pip-wheel-CujgH6
  Running setup.py bdist_wheel for pyblish-base ...   Destination directory: /tmp/pip-wheel-CujgH6
  Running command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-l17biy/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-CujgH6 --python-tag cp27
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/__init__.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/__main__.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/api.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/cli.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/compat.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/error.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/lib.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/logic.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/main.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/plugin.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/util.py -> build/lib.linux-x86_64-2.7/pyblish
  copying pyblish/version.py -> build/lib.linux-x86_64-2.7/pyblish
  creating build/lib.linux-x86_64-2.7/tests
  copying tests/__init__.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/lib.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_cli.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_context.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_events.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_logic.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_misc.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_plugin.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_simple.py -> build/lib.linux-x86_64-2.7/tests
  copying tests/test_util.py -> build/lib.linux-x86_64-2.7/tests
  creating build/lib.linux-x86_64-2.7/pyblish/vendor
  copying pyblish/vendor/__init__.py -> build/lib.linux-x86_64-2.7/pyblish/vendor
  copying pyblish/vendor/iscompatible.py -> build/lib.linux-x86_64-2.7/pyblish/vendor
  copying pyblish/vendor/mock.py -> build/lib.linux-x86_64-2.7/pyblish/vendor
  copying pyblish/vendor/six.py -> build/lib.linux-x86_64-2.7/pyblish/vendor
  creating build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/__init__.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/_bashcomplete.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/_compat.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/_termui_impl.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/_textwrap.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/core.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/decorators.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/exceptions.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/formatting.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/parser.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/termui.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/testing.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/types.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  copying pyblish/vendor/click/utils.py -> build/lib.linux-x86_64-2.7/pyblish/vendor/click
  creating build/lib.linux-x86_64-2.7/tests/pre11
  copying tests/pre11/__init__.py -> build/lib.linux-x86_64-2.7/tests/pre11
  copying tests/pre11/lib.py -> build/lib.linux-x86_64-2.7/tests/pre11
  copying tests/pre11/test_cli.py -> build/lib.linux-x86_64-2.7/tests/pre11
  copying tests/pre11/test_logic.py -> build/lib.linux-x86_64-2.7/tests/pre11
  copying tests/pre11/test_plugins.py -> build/lib.linux-x86_64-2.7/tests/pre11
  creating build/lib.linux-x86_64-2.7/tests/pre13
  copying tests/pre13/__init__.py -> build/lib.linux-x86_64-2.7/tests/pre13
  copying tests/pre13/test_di.py -> build/lib.linux-x86_64-2.7/tests/pre13
  copying tests/pre13/test_logic.py -> build/lib.linux-x86_64-2.7/tests/pre13
  copying tests/pre13/test_plugin.py -> build/lib.linux-x86_64-2.7/tests/pre13
  creating build/lib.linux-x86_64-2.7/pyblish/plugins
  copying pyblish/plugins/collect_current_date.py -> build/lib.linux-x86_64-2.7/pyblish/plugins
  copying pyblish/plugins/collect_current_user.py -> build/lib.linux-x86_64-2.7/pyblish/plugins
  copying pyblish/plugins/collect_current_working_directory.py -> build/lib.linux-x86_64-2.7/pyblish/plugins
  creating build/lib.linux-x86_64-2.7/pyblish/icons
  copying pyblish/icons/logo-32x32.svg -> build/lib.linux-x86_64-2.7/pyblish/icons
  copying pyblish/icons/logo-64x64.svg -> build/lib.linux-x86_64-2.7/pyblish/icons
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/__init__.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/__main__.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/api.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/cli.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/compat.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/error.py -> build/bdist.linux-x86_64/wheel/pyblish
  creating build/bdist.linux-x86_64/wheel/pyblish/icons
  copying build/lib.linux-x86_64-2.7/pyblish/icons/logo-32x32.svg -> build/bdist.linux-x86_64/wheel/pyblish/icons
  copying build/lib.linux-x86_64-2.7/pyblish/icons/logo-64x64.svg -> build/bdist.linux-x86_64/wheel/pyblish/icons
  copying build/lib.linux-x86_64-2.7/pyblish/lib.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/logic.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/main.py -> build/bdist.linux-x86_64/wheel/pyblish
  copying build/lib.linux-x86_64-2.7/pyblish/plugin.py -> build/bdist.linux-x86_64/wheel/pyblish
  creating build/bdist.linux-x86_64/wheel/pyblish/plugins
  copying build/lib.linux-x86_64-2.7/pyblish/plugins/collect_current_date.py -> build/bdist.linux-x86_64/wheel/pyblish/plugins
  copying build/lib.linux-x86_64-2.7/pyblish/plugins/collect_current_user.py -> build/bdist.linux-x86_64/wheel/pyblish/plugins
  copying build/lib.linux-x86_64-2.7/pyblish/plugins/collect_current_working_directory.py -> build/bdist.linux-x86_64/wheel/pyblish/plugins
  copying build/lib.linux-x86_64-2.7/pyblish/util.py -> build/bdist.linux-x86_64/wheel/pyblish
  creating build/bdist.linux-x86_64/wheel/pyblish/vendor
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/__init__.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor
  creating build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/__init__.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/_bashcomplete.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/_compat.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/_termui_impl.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/_textwrap.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/core.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/decorators.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/exceptions.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/formatting.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/parser.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/termui.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/testing.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/types.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/click/utils.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor/click
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/iscompatible.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/mock.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor
  copying build/lib.linux-x86_64-2.7/pyblish/vendor/six.py -> build/bdist.linux-x86_64/wheel/pyblish/vendor
  copying build/lib.linux-x86_64-2.7/pyblish/version.py -> build/bdist.linux-x86_64/wheel/pyblish
  creating build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/__init__.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/lib.py -> build/bdist.linux-x86_64/wheel/tests
  creating build/bdist.linux-x86_64/wheel/tests/pre11
  copying build/lib.linux-x86_64-2.7/tests/pre11/__init__.py -> build/bdist.linux-x86_64/wheel/tests/pre11
  copying build/lib.linux-x86_64-2.7/tests/pre11/lib.py -> build/bdist.linux-x86_64/wheel/tests/pre11
  copying build/lib.linux-x86_64-2.7/tests/pre11/test_cli.py -> build/bdist.linux-x86_64/wheel/tests/pre11
  copying build/lib.linux-x86_64-2.7/tests/pre11/test_logic.py -> build/bdist.linux-x86_64/wheel/tests/pre11
  copying build/lib.linux-x86_64-2.7/tests/pre11/test_plugins.py -> build/bdist.linux-x86_64/wheel/tests/pre11
  creating build/bdist.linux-x86_64/wheel/tests/pre13
  copying build/lib.linux-x86_64-2.7/tests/pre13/__init__.py -> build/bdist.linux-x86_64/wheel/tests/pre13
  copying build/lib.linux-x86_64-2.7/tests/pre13/test_di.py -> build/bdist.linux-x86_64/wheel/tests/pre13
  copying build/lib.linux-x86_64-2.7/tests/pre13/test_logic.py -> build/bdist.linux-x86_64/wheel/tests/pre13
  copying build/lib.linux-x86_64-2.7/tests/pre13/test_plugin.py -> build/bdist.linux-x86_64/wheel/tests/pre13
  copying build/lib.linux-x86_64-2.7/tests/test_cli.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_context.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_events.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_logic.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_misc.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_plugin.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_simple.py -> build/bdist.linux-x86_64/wheel/tests
  copying build/lib.linux-x86_64-2.7/tests/test_util.py -> build/bdist.linux-x86_64/wheel/tests
  running install_egg_info
  running egg_info
  writing pyblish_base.egg-info/PKG-INFO
  writing top-level names to pyblish_base.egg-info/top_level.txt
  writing dependency_links to pyblish_base.egg-info/dependency_links.txt
  writing entry points to pyblish_base.egg-info/entry_points.txt
  reading manifest file 'pyblish_base.egg-info/SOURCES.txt'
  writing manifest file 'pyblish_base.egg-info/SOURCES.txt'
  Copying pyblish_base.egg-info to build/bdist.linux-x86_64/wheel/pyblish_base-1.6.1-py2.7.egg-info
  running install_scripts
  creating build/bdist.linux-x86_64/wheel/pyblish_base-1.6.1.dist-info/WHEEL
done
  Stored in directory: /home/alan/.cache/pip/wheels/73/0b/4a/a1390dff131dc9d7c44199ad258e63ade97852fe57bcd64866
  Removing source in /tmp/pip-req-build-l17biy
Successfully built pyblish-base
Installing collected packages: pyblish-base

  changing mode of /home/alan/.local/bin/pyblish to 777
Successfully installed pyblish-base-1.6.1
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-2VsyFl'

Quite a bit different, eh? Look at all that IO! :o It took your tar.gz and processed it into a wheel for you and then installed it.

In generating that wheel on-the-fly, it put it in the pip cache directory (/home/alan/.cache/pip/wheels for me) so if something else requests it (say another virtualenv), it will be ready to be deployed quickly without regenerating the wheel. Had it been a wheel from the beginning, it gets cached too but you save on not generating it.

Have you experienced problems with the current source distribution that prompted or would be solved by a wheel?

"Problems" is a strong word. Slightly annoying slowness is more like it, lol, but allow me to elaborate exactly why it's extra bad on my end...

At my workplace we have a customized fork of https://github.com/nerdvegas/rez/ with support for a python_requires declaration where we can list our pip package requirements for a given Rez package. (I'm guessing you've heard of Rez before? If not, it's basically a package manager that builds an environment on the fly from released packages sourced from the network.)

Our custom Rez reads all the pip requirements from all Rez packages in the requested rez-environment, "hashes" the request to a string, checks if a virtualenv exists under that hash name and if not it creates one, then activates it and pip-installs all the packages into it. If it exists, it just activates it and that's it. This is quite a handy workflow as we can source pip requirements directly from pypi, but there is one downside in our implementation design, which is that when the hash changes, which would happen as soon as any of the packages' versions are different (when a new version of something is released) then the whole temp virtualenv is thrown out and a new one generated, and pip proceeds to install everything yet again. In other words, we pip-reinstall a lot.

Pip can keep a local cache of recent wheels which helps when re-requesting something and we try to further alleviate things by proxying pypi at the studio to pass through a product called Nexus Repository which acts as a local pypi server with automatic mirroring of the official pypi, autoupdates, caches requested packages and allows us to upload our own packages/wheels to the "studio repo" while coexisting with official pypi packages, which is neat.

That said, pip will reinstall stuff a lot, so for us having wheels for everything is extra important. I could generate some at the studio manually for all pyblish modules, but I figured I'd rather help contribute changes here so that the official releases has good universal wheels for everyone else to benefit too. :)

Am I making more sense now?

darkvertex commented 6 years ago

Hmmm, just spotted pyblish-qml was not making a universal wheel. Here's the PR to fix that.

mottosso commented 6 years ago

Thanks for taking the time to include your research and reasoning!

I generally don't accept anything into the codebase unless it solves a specific problem. In this case I couldn't quite figure out what that problem was, until:

In other words, we pip-reinstall a lot.

Which seems like the key issue here; you need faster re-installs. If that's right I'd be happy to give this a try, let me know when you're happy with this PR and I'll make a release out of it. Preferably we'd try the new mechanism with one project at a time, till we're sure it solves your problem.

darkvertex commented 6 years ago

Which seems like the key issue here; you need faster re-installs.

Yes, exactly right! 👍

If that's right I'd be happy to give this a try, let me know when you're happy with this PR and I'll make a release out of it.

I think it's good now. I just added a tiny last change that makes the LICENSE.txt get bundled inside the generated wheel.

Preferably we'd try the new mechanism with one project at a time, till we're sure it solves your problem.

Sounds good to me. I guess we can try releasing this repo first and if it "works" and pypi ends up with proper universal wheels and pip installs faster, we can do the other PRs.

mottosso commented 6 years ago

Thanks!

I've made a release, but Travis appears to have undergone backwards breaking changes.

https://travis-ci.org/pyblish/pyblish-base/jobs/425621332#L748

I don't have time to look into this at the moment; if you want to help, then you can investigate why this happens. Putting this on my list of things to look into.

darkvertex commented 6 years ago

Weird. I will try to research why that happened.

mottosso commented 6 years ago

Thanks Alan, appreciate that!

darkvertex commented 6 years ago

Hmmm, do you think it might be because of the python: 2.6 bit in the deploy: block?

Higher in the file:

python:
- 2.7
- 3.5

No 2.6 there and so the first "runtime" running first is 2.7, so I think the error means "I can't release because I'm not on the 2.6 runtime." -- Why did it work before? I don't know, but maybe their wheel builder mechanism checks things more strictly.

In any case 2.6 hit EOL (End of Life) in late 2013, right? So no harm in bumping the deploy runtime to 2.7. I made a PR for this: #339

mottosso commented 6 years ago

Most interesting, let's give this a try!

mottosso commented 6 years ago

I think we've got a winner, are you able to confirm?

https://pypi.org/project/pyblish-base/1.7.2/

darkvertex commented 6 years ago

It works and it's fast!

> pip install -v --user pyblish_base-1.7.2-py2.py3-none-any.whl
Processing ./pyblish_base-1.7.2-py2.py3-none-any.whl
Installing collected packages: pyblish-base

  changing mode of /mnt/users/alan/.local/bin/pyblish to 755
Successfully installed pyblish-base-1.7.2
Cleaning up...

(That's all the prints, and that's even in -v verbose mode!)

I timed it:

Wheel took 3.83s to install.
Tar.gz (source distribution) took 8.30s to install.

If my math is correct, that's 53% faster to install. :sunglasses:

darkvertex commented 6 years ago

The other PRs are ready for when you wish to merge them:

pyblish-lite: https://github.com/pyblish/pyblish-lite/pull/105 (✔️ merged)
pyblish-qml: https://github.com/pyblish/pyblish-qml/pull/298 (✔️ merged)
pyblish-maya: https://github.com/pyblish/pyblish-maya/pull/74 (✔️ merged)
pyblish-houdini: https://github.com/pyblish/pyblish-houdini/pull/17 (✔️ merged)
pyblish-nuke: https://github.com/pyblish/pyblish-nuke/pull/41 (✔️ merged)

I checked we didn't have another "impossible runtime" issue and made sure the license file was included in the setup.cfg like I did here.

pyblish / pyblish-base

Edit Travis config to build a Python wheel (.whl) along with the .tar.gz file. #338