indygreg
commented
3 years ago I believe the feature you are requesting is possible today. However, it may not be obvious and PyOxidizer's config file may not support exactly what you want.
The OxidizedFinder meta path finder exposes some APIs to Python that allow it to read and write packed resources data files/blobs. This is described at https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_freezing_applications.html. The underlying Rust code doing the work is used by both PyOxidizer and the oxidized_importer Python package.
From PyOxidizer, the PythonEmbeddedResources (https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_config_type_python_embedded_resources.htm) Starlark type represents a collection of Python resources. If you define a target function that returns an instance of this type, one of the files written will be loadable into OxidizedFinder.
For what it's worth, I have aspirations of formalizing the custom resource serialization format and proposing it as an alternative mechanism for distributing Python packages. I think the value proposition of immutable single-file distributions of Python packages that are more performant than wheels is compelling. Unfortunately, the data format and features within need a lot of work before that can be considered. But the foundation for building your own alternative Python resources distribution format is definitely there in the code today!
I'd like to encourage more experimentation in this space. If there's any APIs or functionality that you'd like to see, please suggest ideas!
EndilWayfare
In a typical Python scientific computing project, the on-disk size is probably dominated by large dependencies like
scipyandnumpy(88 and 46 MB in my build, respectively), not to mention the standard library. These dependencies likely change at a far slower rate than the local Python and Rust code that makes up the rest of the project. Having to redistribute a 100 MB+ bundle every update is less than ideal.Granted, those large dependencies are also
__file__users, and must be stored in a "lib" folder anyway. For now, it's possible to leave those alone and distribute the smaller binary (39 MB in my build) instead. However, that binary still contains the stdlib and other packages that can be classified, because I'd like to make use of the fast, zero-copy imports where I can. In future, once the package resources problem is (more) resolved, (more of) those dependencies will make it into the binary to enjoy the same benefits. This is key, I think: the benefit from packing dependencies into the binary is not just "single file" distribution.It would be really neat if we could package rarely-changing dependencies into a "dependencies.dll" file, or similar, to preserve fast imports while reducing update size. Dynamic linking being what it is, I imagine that this would be very nontrivial to implement cross-platform. Or maybe not; this is outside my area of expertise. Just wanted to raise it as a suggestion!