Include all wheels from a directory

[adferrand]

Let's say I want to create an installer from an existing application. This application is complex, with a lot of dependencies, various packages, and its build is configured "by the book", with a setup.py that declares correctly all this stuff. The application may or may not be published on PyPi, with or without wheels. But with the project source code, as it is correctly configured, I can use pip wheel . -w my_wheels_folder to generate in my_wheels_folder all the wheels required: the application itself, and all its dependencies at their good version.

So now, it is required to include all necessary wheels in the installer, at their good versions as defined in setup.py. For now, I can see only two ways to do that:

• either constructing on the fly an installer.cfg that declares in pypi_wheels all required wheels, building them using pip, and set extra_wheel_sources to be the directory given to pip wheel.
• use pynsist --no-makensis to extract manually wheels to build/nsis/pkgs before invoking makensis

Either way, there is not straight forward mechanism to say to pynsist "take all the wheels that I put here, and extract them into pkgs".

So should this be in the scope of pynsist ? I believe so, as it is not a build issue (pynsist do not handle wheels build), but a packaging issue for the windows installer. And it would ease drastically a process that I think to be quite common.

If it is in the scope, I am willing to do the implementation. I think of two ways:

• adding a configuration parameter to take unconditionally all wheels from extra_wheel_sources (any wheels declared in pypi_wheels not present in extra_wheel_sources will continue to be gathered from pypi if possible)
• adding a independent configuration parameter, local_wheels, that accepts a list of ant paths to be unconditionally included and extracted in pkgs, like this:

  [Include]
  local_wheels=my_wheels_folder/*.whl
           my_other_wheels_folder/**/*.whl

[takluyver]

A couple more options that will work today:

1. Unpack all the wheels into a pynsist_pkgs directory next to installer.cfg. This is pretty similar to your option 2, except you can run Pynsist as normal, without the --no-makensis flag and separately running makensis afterwards.
2. Skip the config file and run Pynsist from a Python script. It basically amounts to InstallerBuilder(**kwargs).run(), where the keyword args hold the same data as the config file (API docs). It may need a bit of fiddling to work out exactly what form the metadata needs to be in.

This is not to say no to the feature, but I'm interested to hear how attractive either of those options would be for you.

I'm finding there are two different kinds of use case for Pynsist: for smaller projects, you can run pynsist installer.cfg directly and let it take care of everything. For larger projects, like you described, it's a step in a build process, where other steps prepare the input for it. Often that leads to people either using the Python API, or generating the config file on the fly - but I don't want to be overly prescriptive that larger projects must do that.

If I do go for the feature, I prefer the local_wheels variant. I may have been considering that possibility when I picked the name pypi_wheels.

[adferrand]

Thanks for your response ! So, about your propositions.

About 1. I was not aware of the existence of pynsist_pkgs, I did not see in the documentation. It is already better than the two ways I described. But it stays to be re-making what pynsist is already able to do: extract wheels from various source and install them in the right place. Why not reusing a code that has been already strongly tested and deployed ?

About 2. It indeed corresponds to what one could do in a complex project with the need to construct a really fine tuned packaging process. For what I am currently doing, it is essentially making the package process in Python instead of Powershell. That is also an improvement on its own. Really ^^

For the records, you can check by "beautiful" Powershell script for Certbot here: https://github.com/adferrand/certbot/blob/windows-installer/windows-installer/construct.ps1

So each of the proposition improves my current stack, but not as much as including wheels directly by pynsist would do. In a way or another (Powershell script or decompose pynsist processing into a Python script), it reduces a lot the code from one side, and add very little to the side of pynsist. And adding this option offers it widely to any user of pynsist. I think it is really a good improvement for any user trying to package an installer for an advanced application, and I would definitively use pynsist for every Python application I try do port to Windows.

I would also prefer the local_wheels variant. If you are good with it, it can start an implementation.

[takluyver]

OK, I think I'm sold on the idea.

A couple of remaining questions:

1. What happens if you specify the same package twice - either as two different local wheels, or local and on PyPI? I'm inclined to say it should be an error, because "refuse the temptation to guess". It could already happen with pypi_wheels, but it will be easier to run into it accidentally with this.
2. What should it do if there's an incompatible wheel (e.g. wrong platform, wrong Python version) matching the glob? Again, I'm inclined to say it should be an error. If you want to use it with a directory containing a mixture of wheels, you should use extra_wheel_sources and list the specific ones you want in pypi_wheels.

[adferrand]

When you specify wheels through pypi_wheels, you can forgot to specify one required dependency. In this case, pynsist will build a fully operational installer without complains, but you will see that the application installed with the installer fails to start because of a missing module.

And I find this totally understandable, because it is not of the pynsist concern to verify that the build is correct.

So for theses situation, I think indeed that pynsist should not try to guess. And in the extend of few packaging process controls, it should fail fast:

• two times the same wheel library (same or different version, platform, python version): fail
• a wheel library of wrong platform or python version: fail
• something catched for the specified paths but not a wheel: fail

[takluyver]

Sounds good to me. Maybe eventually this will become the recommended way to use Pynsist: use another tool to fetch the wheels you need, and then tell Pynsist to assemble from those.

Some historical context for not automatically finding dependencies: when I started Pynsist the main option was the packages list to find packages by import name. Other tools such as cx_Freeze track imports by static analysis to find dependencies, but I found this unreliable, so I said that for Pynsist, you would have to explicitly list all packages to bundle. I started writing something to identify dependencies by running an application.

When using wheels became an option, that was the background, and automatically resolving dependencies of Python packages is non-trivial, so it stuck. It also has the nice side effect that you have to specify a version for every package to include, making builds less ambiguous. Perhaps a better way to do this would be some kind of lock file; I don't think I knew of that concept at the time, and I'd still find it extra complexity now.

[adferrand]

Indeed. And here with local_wheels you get again potentially non repeatable builds as setuptools allows to specify a version range. In the Java world, it was something Maven decided to remove from their build system. So for a lot of people, the Python dependency stays broken. But at least it is the standard Python way to build applications, and it will be the responsability of maintainers of an application to deal with this, maybe if some existing solutions to make builds predictable.

[takluyver]

Closed by #164.