Added a ZipFileSystem implementation

[UncertainProd]

I added a Zip file system, which can construct a filesystem using a zip file. It only works with uncompressed zip archives at the moment but seems to work on both windows and html5 from what I've tested.

Example Usage:

#if sys
var byts = File.getBytes('mods/MyMod.zip');
var zfs = ZipFileSystem.fromZip(byts, {});

var sp1:FlxSprite = new FlxSprite().loadGraphic(
                                   BitmapData.fromBytes(
                                                 ByteArray.fromBytes(
                                                                zfs.getFileBytes('MyMod/images/icons/icon-pico.png')
                                                 )
                                        )
                               );
sp1.setGraphicSize(100, 100);
add(sp1);
#end

I had also added some functionality to make an uncompressed zip from a directory, but it is still unfinished as of now so I removed it. Most zip programs have options to make uncompressed zip files either way.

[EliteMasterEric]

Thank you very much for taking the time and effort to submit this! I appreciate that.

This PR is interesting but only part of what I'm looking for in this kind of system.

With the current implementation, the entire contents of the zip are loaded into memory when the mod is loaded. This may be fine for smaller mods but becomes problematic when mods with large textures are involved.

I would like to see an implementation of this that uses the central directory file header of the ZIP, see here. This would allow you to access the list of files by seeking to the end of the ZIP and reading the list of files, then reading files by fetching specific parts of the container.

This would allow the file system to be more efficient, since only files the program requests would be extracted.

I looked into this previously, and I've seen this in other languages implementations of ZIP but not in Haxe.

[UncertainProd]

Hey, thank you for the reply! I agree that using the central directory file header would be better, so only the bytes that are requested will be loaded into memory. While I think it should be possible to implement that for sys targets, I can't seem to find a way of avoiding storing the entire zip in memory when it comes to html5 targets. If I'm not mistaken, it looks like a file uploaded from a user through the browser file upload will be entirely read and stored in memory by the browser. So far I haven't seen a way to work around that.

[EliteMasterEric]

Hey, I think this is a fundamental limitation of HTML5. One optimization that can be done is to remove the ZIP from memory once it's files are extracted, but aside from that, I think we just have to accept that any uploaded files must remain in memory.

Some other notes:

• You mentioned that Polymod functions fully with mods on HTML5? That's good to know, I had not fully tested that yet. Let me know if there are any issues.
• Have you tested the behavior when multiple mods placed in the same zip file? This would be quite useful for the purposes of bringing back the modpack feature I removed in a previous patch.

On Wed, Jul 13, 2022, 16:14 UncertainProd @.***> wrote:

Hey, thank you for the reply! I agree that using the central directory file header would be better, so only the bytes that are requested will be loaded into memory. While I think it should be possible to implement that for sys targets, I can't seem to find a way of avoiding storing the entire zip in memory when it comes to html5 targets. If I'm not mistaken, it looks like a file uploaded from a user through the browser file upload will be entirely read and stored in memory by the browser. So far I haven't seen a way to work around that.

— Reply to this email directly, view it on GitHub https://github.com/larsiusprime/polymod/pull/127#issuecomment-1183634366, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDLVRRNJ45TBJOXX3UILW3VT4PRZANCNFSM53FCA7PA . You are receiving this because you commented.Message ID: @.***>

[UncertainProd]

• You mentioned that Polymod functions fully with mods on HTML5?

I didn't test it out completely, but this is what I did on html5 for testing the loading of the zip file (in flixel). So far all I tried was loading the bytes of an image in the zip onto an FlxSprite.

import flixel.FlxSprite;
import flixel.FlxState;
import flixel.util.FlxTimer;
import haxe.io.Bytes;
import openfl.display.BitmapData;
import openfl.events.Event;
import openfl.net.FileFilter;
import openfl.net.FileReference;
import openfl.utils.ByteArray;
import polymod.fs.ZipFileSystem;

class PlayState extends FlxState
{
    var _download_fileref:FileReference;
    var daBytes:Bytes;

    var zfs:ZipFileSystem;

    function loadZip()
    {
        trace('starting file reference');
        _download_fileref = new FileReference();
        _download_fileref.addEventListener(Event.SELECT, (e)->{ _download_fileref.load(); });
        _download_fileref.addEventListener(Event.COMPLETE, onLoadComplete);
        trace('browse starting');
        _download_fileref.browse([ new FileFilter("Zip files", "*.zip") ]);
        trace('browse over');
    }

    function onLoadComplete(e:Event) {
        daBytes = getHaxeBytes(_download_fileref.data);
        _download_fileref.removeEventListener(Event.SELECT, (e)->{ _download_fileref.load(); });
        _download_fileref.removeEventListener(Event.COMPLETE, onLoadComplete);
        _download_fileref = null;

        zfs = ZipFileSystem.fromZip(daBytes, {});

        var bmpf = BitmapData.loadFromBytes(
            ByteArray.fromBytes(
                zfs.getFileBytes('ThirdMod/images/icons/icon-pico.png')
            )
        );

        bmpf.onComplete((bmp)->{
            var sp1 = new FlxSprite().loadGraphic(bmp);
            sp1.setGraphicSize(100, 100);
            add(sp1);
        });
    }

    override public function create()
    {
        super.create();
        // Polymod.init({ modRoot: 'mods', errorCallback: (e)->{ trace('PolymodError: ${e.message}'); } });
        new FlxTimer().start(0.5, (tmr)->{
            loadZip();
        });
    }

    function getHaxeBytes(b:ByteArray)
    {
        // not sure how else to convert a ByteArray into haxe.io.Bytes :/
        var le_bytes = Bytes.alloc(b.length);
        for(i in 0...b.length)
        {
            le_bytes.fill(i, 1, b.readByte());
        }
        return le_bytes;
    }

    override public function update(elapsed:Float)
    {
        super.update(elapsed);
    }
}

• Have you tested the behavior when multiple mods placed in the same zip file?

In the setup that I did above, it does seem to work when multiple mods are zipped into a single zip file

[EliteMasterEric]

I didn't test it out completely, but this is what I did on html5 for testing the loading of the zip file (in flixel).

The problem I see with this is that you are directly accessing a specific mod's file by name through the Polymod file system. This is not how Polymod is intended to be used. You should be attempting to use the normal file retrieval functions of OpenFL/Flixel (or whatever the engine you're using is) and attempting to access a file as though it was included in your applications assets folder at build time. Polymod does the rest.

In the setup that I did above, it does seem to work when multiple mods are zipped into a single zip file

This is the other issue. I would like to see multiple mods working.

Here is how I would like to see this function:

• Import a zip, then query the list of files for any _polymod_meta.json files.
• Based on these, add files to the in-memory map such that their paths look like mods/ModId/_polymod_meta.json, then put any adjacent files or directories in that mod folder.
• If the _polymod_meta.json is at the root of the zip, make the mod ID the file name; otherwise, make it the folder name.

This should allow for either a mod ZIP directly containing the required files, or a zip containing multiple folders each with their own mod. In the future, functionality to ensure mods load together can also be added.

Another change I'd like to see is a move away from an in-memory map. The file system should read the file from the zip when requested, and allow the host application to ensure proper caching and cleanup. That central directory thing is important for this to work.

There should be a second file system, MemoryZipFileSystem, which acts like the current system does, reserved only for use on web where the zip can't easily be re-accessed.

I would also like to see the non-memory file system support a mix of zipped and non-zipped mods.

[UncertainProd]

I managed to get the flixel example working on html5 with the in-memory zip file system. It also supports multiple mod folders zipped into one zip file. The only difference in the web version is that the images need to be loaded asynchronously (using Assets.loadBitmapData instead of Assets.getBitmapData).

[EliteMasterEric]

This seems great! It doesn't have the central directory optimization I was looking for but we can add that in a future change.

Once I get on the computer I'm going to move your sample to flixel-zip (so the old one isn't overridden) and merge it.

[UncertainProd]

Alright, thanks!

[UncertainProd]

Okay, so I made a separate zip-filesystem for sys targets that doesn't store the entire zip in memory, but reads the required data whenever getFileBytes() or getFileContent() is called, and I replicated the flixel sample to showcase this SysZipFileSystem

[EliteMasterEric]

Holy shit ZipParser is goated thank you so much for figuring that shit out

I'm gonna get this merged today so I can test, do some cleanup, etc