Packaged Standard Library missing modules

[tmontes]

Fact

While exploring #381, trying to package the Mu Editor on Windows, I realized that the packaged Python environment is missing modules from the Standard Library.

In particular I realized that tkinter, pip, and venv are missing.

Then I tried the same on macOS and confirmed that they're missing too.

Question

While I do understand that by creating and using an alternate support package these limitations can be overcome, I wonder if, from a user/packager experiece, it wouldn't be desirable for the packaged environment to be as capable as the environment in which the original code is developed and tested and, thus, have it be that way by default?

Thought

(warning: crazy idea ahead) :)

I wonder if instead of using a pre-built support package briefcase could instead "copy and adjust" the current Python installation into the packaging src directory. Maybe some adjustments might be needed to ensure the "copied" environment is relocatable and works nicely, but how hard could that be? (grin)

Of course, for some target platforms that is not an option given that briefcase is packaging with a foreign target like iOS targets being packaged on macOS, for example.

Thoughts?

[freakboy3742]

In the windows case, the interesting detail is that we're using the stock "Windows x86 embeddable zip file" provided by Python.org. I'm guessing tkinter is omitted because the extra DLLs needed; pip and venv are omitted because the "embedded" use case shouldn't (generally) need to install extra code or set up a new environment.

The macOS (and iOS, and Android) builds that Briefcase uses are all managed by BeeWare; we've mostly followed the lead of the Windows installer in terms of omissions. In fact, having pip in the iOS binary could actually constitute grounds for ejection from the iOS App Store, as they prohibit using any mechanism to get around App Store code review.

That said: I completely agree that customizing the standard library is one of the big issues facing Briefcase.

"Provide everything" is obviously a safe default. However, it's also not a good solution - it means you end up with support packages (and thus apps) that are a lot bigger than they need to be.

I have a very slightly selfish desire to avoid packaging tkinter because it's a competitor to Toga (and my problems with Tkinter are why Toga exists); but fundamentally, I don't have any good reason why Briefcase shouldn't support tkinter.

pip and venv are another matter; there's definitely a subset of use cases (and, admittedly, Mu is one of them) for which including pip and venv make sense; but for the default "user application" case, they're "overhead with security implications".

"Relocating" an existing system install isn't an easy option, because of dependencies on external DLLs.

So - we're left with the problem of how to make it easy for end users to build and use custom support packages. This is currently supported in Briefcase by allowing you to define the support package that you want to use; but that leaves the task of actually building the support package to the end user.

We could invest in building tooling to manage that process; but the better fix might actually be to go the other direction. There's a been a little bit of discussion on the Python core team about a "Minimum Viable Python" distribution - a Python distribution that contains literally nothing in the standard library, with all the standard library specified by as dependencies. You want httplib? tkinter? pip? venv? Include it in your requirements file. There's obviously detail to sort out here, but it has the benefit of (a) being explicit, and (b) ensuring that support packages are as small as they plausibly can be - which is a benefit for the application distribution use case.

[tmontes]

@freakboy3742,

Thanks for sharing your detailed thoughts. They are pretty much inline with what I imagined could be your ideas -- having myself been in tune with your work on BeeWare for a while now -- and also very much aligned with the "BeeWare vision", if I may call it so. Consistency is important.

I'm coming from a different background with a (maybe?) different vision of a Briefcase with a "life of its own", without betraying the needs/purposes of BeeWare -- could it be(come) the holy grail of cross-platform (and cross-target?) Python application packaging? Will there be a day when a user can just say "briefcase, go!" and have their application automatically packaged and distributed to "all platforms", regardless of their development platform? We can dream... And then we could do something, like you have been doing. :-)

I have used many Python packaging tools in the past and I like the direction Briefcase is taking, more and more (not sure about pyproject.toml, but ultimately that's an implementation detail and a whole other conversation). Congratulations + Credits + Thanks a lot to you are in order! :-)

The ability to handle arbitrary support packages is very valuable and, like you point out, lets users adjust -- the challenge in building such support packages is a whole different story. Interestingly, I just found out the Python Standalone Builds project and took the macOS variant for a quick spin (a bit reckless, downloading a running an arbitrary binary, but I publicly assume that I did just that). The image size is reasonable (~100MB) for an all batteries included package (tkinter, ssl, venv, pip, all included -- fudamental Mu requirements) and, after trivial trimming down, I brought it down to ~35MB. Not bad! While I understand the need to have custom Python builds for iOS/Android, I wonder if the efforts associated to maintaining the common platform variants (say, win/lin/mac) under the BeeWare umbrella could be merged with or, maybe better, leverage that project's work... (were you aware of it?)

I have heard of the "Minimum Viable Python" yes, thanks. That may be the future and it may even be a good idea -- I have not given it much thought, yet. But people like you and me want to do things today-ish... :-)

PS: I think I'll kick off a mini-tkinter project (no objections to Toga at all, but wanting to work within the Standard Library) to serve as an "exerciser" to both briefcase and the ability to integrate Python Standalone Builds as a support package.

[tmontes]

More PS: One important aspect I forgot mentioning, regarding the ability to "copy and relocate" the current Python installation is, more than non-system shared lib dependencies -- which can also be copied and relocated -- the matter of distribution. One may not be allowed to do that!

But the possiblity of running Briefcase fully offline from the start is something I myself find potentially valuable, even if not easy to achieve.

[freakboy3742]

On the subject of handling non-shared system libs: The AppImage target on Linux actually already does this. Part of the packaging process is inspecting binaries to determine libraries that need to be linked, copying them into the bundle, and rewriting them to reflect their new location. That approach seems to work reasonably well for Linux (well enough that an AppImage built on Ubuntu 16.04 will run under Fedora 29). So - it can be done on some platforms, and it may be possible on others (see #371 and #372 for some possible options); it's just complex.

As for running Briefcase completely offline; I completely agree that is desirable. I spend enough time on planes that this would be a useful feature to have. I know it's not currently possible because the support package download needs to be online to determine the release, but I'd definitely consider that a bug (#388).

I'll leave this ticket open as a marker for the fact that we need a generic fix for customizing support requirements. In that context, I consider venv, pip and tkinter to be completely fair game for the packaging process. Python-Standalone-Builds might be a good project to collaborate with to that end. And +1 to any effort to package tkinter as a standalone - I'd love to be able to offer Tkinter as an option in the default Briefcase template.

[rmartin16]

In that context, I consider venv, pip and tkinter to be completely fair game for the packaging process. Python-Standalone-Builds might be a good project to collaborate with to that end. And +1 to any effort to package tkinter as a standalone - I'd love to be able to offer Tkinter as an option in the default Briefcase template.

Out of curiosity, in https://github.com/beeware/briefcase/issues/318#issuecomment-2254221505, I explored how I could use tkinter with Briefcase; in short, use a Standalone Python and update the stub for the different filesystem layout of Python. This Python also brings along venv and pip.

It may be reasonable to move all of Windows builds from the python.org embedded Python to using Standalone Python; that would also align with Flatpak and AppImage builds.

[freakboy3742]

I'm not sure how I feel about switching to standalone Python for Windows by default.

To me, using the officially distributed "Windows embedded" binaries is an aspirational goal - it's something we would do for macOS, Linux, iOS and Android if only the required artefacts were available (which - for the record, is something I'm hoping to be working on the for the 3.14 release cycle, at least for macOS, and possibly for iOS and Android). Moving away from the official package for Windows is not what I would consider "progress".

On top of that, the Standalone package is twice the size of the official Embedded package - I'm guessing Tkinter is a good chunk of that. This is content that Toga, Qt, PPB, Pygame, Rich and any other GUI toolkit users would need to strip in order to make a space-efficient app. Briefcase has the options to do this, but it's not trivial; and it's extra work that each of these toolkits would need to do, just so that Tkinter users can be supported.

That said - I acknowledge that Tkinter usage is non-zero, and it would be good to be able to support Tkinter projects in Briefcase. To that end, I'd be a lot more amenable to taking the pieces that make tkinter work in Standalone, and packaging them as a third party package on PyPI. It should even be possible to extract the binary artefacts directly from the Standalone package, so that tkinter is Just Another GUI Toolkit from the perspective of Briefcase. That would also align with another long-term project that I'd like to see in CPython - a "no batteries" distribution, where any standard library module that isn't literally required to start the interpreter must be explicitly installed.

Alternatively, I could be convinced to make Standalone usage an option. It's sufficiently standardised that I it's not completely unreasonable to make support_package = 'standalone' (or something similar) a valid configuration option which would then apply the required changes. I'm not sure this would ultimately be any easier than packaging Tkinter (since, based on the writeup you provided, the package layouts are quite different), but it would be another way forward.