DEFLATE / PK3 / PAK / ZIP support, both for importing files and messing with assets on the fly.

RockstarRaccoon commented 4 days ago

Is your feature request related to a problem? Please describe.

The DEFLATE algorithm is popular for compression, as it gives good compression with fast decompression times, meaning a lot of formats use it, including zip files. There are a large number of files out there which are basically just another file run through the deflate algorithm, and a large amount of assets left on the open web as a zip file filled with other files.

The relevant example that made me think of this was trying to make a K Record out of a set of Dead Milkmen recordings that Joe Jack Talcum made available on his website: they are free to download, but only as MP3s inside of a zip file. Resonite supports the lone MP3 files, but not the zip of them.

Another example is the .PK3 format, first used for Quake, and later used by updates to the original Doom engine as a successor to the .WAD format.

Another example is some resonite assets themselves, which, as I understand, use the DEFLATE algorithm until they are loaded by Resonite. (I hope)

Describe the solution you'd like

I would like a set of ProtoFlux components...

DEFLATE - Runs the deflate algorithm on whatever data stream is put into it, preferably with some arguments like encryption.
UNDEFLATE - runs the decompression algorithm on whatever compressed data stream is put into it, once again, preferably with some arguments.
LoadZIP - treats whatever file is being stuck into it as a ZIP file, loading it out as individual files (somehow?)
PackZIP - takes a set of assets and puts them in a single zip file, which I guess could be popped out into the world or something.

Describe alternatives you've considered

It has occurred to me that someone could theoretically implement DEFLATE through existing Protoflux, but this would not only be a painstaking task, but seems unnecessary considering y'all already have support for DEFLATE and likely ZIP loading somewhere in the program code.

Additional Context

The actual zip files may need to wait until arrays are supported, but I think we could still use raw DEFLATE for a few things. It has occurred to me that if it gives the files in some other format, I could make something that drops them into a set of slots. I also have that virtual computer system I've been working on that I would like to support CSV and RTF or HTML files within, and I think I could use this to make external storage simpler.

Requesters

Nuki

Frooxius commented 4 days ago

There's a lot of stuff in this issue and I'm a bit confused on what exactly you're actually looking for, because some of these are very different things.

First, DEFLATE and ZIP are quite different things from technical perspective. DEFLATE is a compression you can apply on data stream to produce a compressed data stream. That can be useful for custom data streams, but this won't help you working with ZIP files as much. ZIP itself is a container that has file-system-like structure and can have a number of DEFLATE streams inside of it.

Each of these has different kinds of applications and depending on what you want to do, one might be more useful than the other.

DEFLATE also doesn't support encryption, so that's not something we can support as part of this.

But you're also asking for nodes for working with files - we don't really have any file IO support in the first place and that's something we'd be EXTREMELY careful about adding, as there's severe security risks.

Resonite assets themselves mostly don't use DEFLATE. They use either LZ4 or LZMA, which are different compression algorithms. But I am confused on what exactly are you asking for there actually? Are you wanting to decode them manually yourself?

From the initial description it sounds like you just want to be able to import MP3 files that are in a ZIP archive easily? But the ProtoFlux nodes won't really help you with that.

One thing we can add (and that's actually planned) is to be able to browse ZIP and other archives in the file browser as folders and import files directly, without having to unpack them first externally.

But I'm confused if that's what you actually need.

I guess my question is - what is the problem you're trying to solve? What is the end goal that you hope this will achieve, without focusing on any of the technical details?

EDIT: I also haven't touched on stuff like PK3/PAK and such - these might be built around ZIP files, but they also contain a lot of special proprietary data. I don't understand what you actually want Resonite to do with these files? Do you just want to import the raw binary data from them? Do you expect Resonite to support this internal data in some way?

XDelta commented 4 days ago

Looks like pk3 files have some support in assimp although there are tons of ways that can be structured and I don't know off hand if there is any currently supported forms of that for reso. Looks like importing them just crashes the game.

Frooxius commented 4 days ago

Assimp doesn't give you the raw files inside the file, but it'll import the actual model and try to reinterpret the data. Which seems like what most people would want, but since this is lumped with this issue, which seems more about the raw access, I'm unsure.

That's part that I'm mainly confused by - is the request to be able to import the 3D model data from PK3? Or just browse the raw files as a ZIP archive? Those are very different requests.

RockstarRaccoon commented 3 days ago

To clarify on your question, my end goal is to be able to pull a zip file from the open internet and access assets inside. The example I gave at the beginning was a zip file I wanted to use for a KRecord which has MP3s in it.

Frooxius commented 3 days ago

What do you mean by "access" exactly?

From the solution you're asking for it sounds like you want to work with the raw data streams of those ZIP files with ProtoFlux.

But just reading the problem it sounds like you just want to import the files out of the ZIP archive without having to extract it in advance.

Those are very different problems, with very different solutions and it's not clear to me which one you're after.

RockstarRaccoon commented 1 day ago

You know, the problem might be that I'm thinking of all this as being related, and maybe it's not as related as I think. That's why I put it in a single request.

I would like to be able to get assets out of a zip file that contains them without having to unzip it beforehand.

I would also like to be able to use DEFLATE on raw data streams and raw files within Resonite to compress and decompress said data.

These may actually have been better put as two separate requests.

Yellow-Dog-Man / Resonite-Issues