Re-implement zip apps (standalone bundles) to easily distribute binary releases for Lua apps

[rdw-software]

Similar to what evo-luvi (and luvi) can do, though I remember there were quite a few kinks that needed to be ironed out.

See also: https://docs.python.org/3/library/zipapp.html

Goals:

• [x] Can use a CLI command (e.g., evo build) to create a zip archive that runs as a standalone app
• [x] When run from inside a zip app (i.e., bundled app), the runtime loads the main.lua entry point

Evo-luvi (and luvi itself) have some extra options like setting a different entry point. I'd rather avoid that, at least for now.

[rdw-software]

I remember there were many problems with VFS lookups and path resolution, if trying to load files from the bundle. To avoid that and still enable the most important functionality (loading files from the bundle, without re-architecting the app), I'd consider something like the following to be a decent starting point for further refinements:

• A new import function can be used in scripts (bundled or otherwise)
• This function accepts typical file system paths (no weird require("some.dir") paths)
• If the interpreter is running from a bundle, it will look in the bundle first and the FS second
• Otherwise, it will simply call require after converting the path to a dot-separator format

This would solve multiple problems at once:

• Can use a sane paths notation even in apps that don't need to be bundled
• Can implement relative path lookups later (if needed, as in evo-luvi)
• Can extend to include module paths (.evo folder, as in evo-luvi)
• Can use import everywhere even if the app is not meant to be bundled
• Can simply load files from the bundled app as if they were required

Edit: This would still introduce some complexity and a sort-of "module system", which I'm not a fan of. Maybe it's more sensible to just start with the "always load main.lua from the archive and everything else must be present on disk" approach first?

Also, since main.lua has access to miniz and everything, more complex processing could be implemented as needed (by users).

[rdw-software]

I actually ran into some more problems: The offset computation breaks down for large files (>8MB) in miniz. It seems the Luvit fork modified the code for this in mz_zip_reader_read_central_dir and other parts, which means there are two options:

1. Copy/paste Luvit's fork and don't benefit from upstream changes to miniz itself (but possibly discover new issues)
2. Somehow plaster over the original miniz issues (such as the early exit that breaks INFLATE completely) to hack in the changes

I dislike both of these, so I choose option 3: Screw this and revisit it later, maybe. It's not that important right now, anyway.

[rdw-software]

OK, actually I hate giving up... Let's survey first the actual list of changes (diff between Luvit miniz and upstream):

• The diff is enormous, which may be partially because the upstream has done significant work in the past ten or so years
• This effectively rules out the possibility of integrating both forks (without expending unreasonable amounts of time)

Since using upstream miniz makes it impossible to implement zip apps and using the Luvit fork is putting extra maintenance work on me (or them, if there are bugs in the fork itself or upstream changes need to be integrated), a workaround is needed.

Prototyping some ideas, the following solution might work - but it introduces some extra complexity:

• As appending executable zip-app doesn't work without modifying miniz, introduce a (slightly modified) "zip app" format
• This can then be processed by the runtime on load, the contained files extracted to a temporary directory, and then loaded
• The format should be as simple as possible, so it'll just be adding an extra signature "footer" with the actual sizes/offsets
• From there, it's trivial to load the zip archive in memory (or extract its contents) using just the standard miniz binding

This has been confirmed to work with the current executable size (21MB on Windows, which is beyond the 8MB limit).

Potential advantages:

• Advanced bundling can be implemented on top of this format, if ever needed (though I'm not a fan of adding complexity)
• This approach decouples the zip app processing/packaging from implementation details such as miniz's directory seeking code
• Reading and writing the signature is trivial with LuaJIT's FFI, similar to any other binary format with a fixed layout

Now, to the disadvantages:

• Packaging and especially unpacking apps becomes more complicated due to having to deal with the signature first
• Regular zip tools, like 7zip or unzip, may not be able to deal with it (unzip still works, and 7zip didn't before anyway)
• The file format must be maintained and documented properly as it's even more arcane than "just" appending (which is OK)

So overall it seems like a workable solution, albeit not an ideal one. The rest depends on how much should be added on top.