jkent / frogfs

Fast Read-Only General-purpose File System (ESP-IDF and ESP8266_RTOS_SDK compatible)
https://frogfs.readthedocs.io
Mozilla Public License 2.0
25 stars 32 forks source link

[WIP] Add zopfli as default program for deflate algorithm #64

Open X-Ryl669 opened 1 month ago

X-Ryl669 commented 1 month ago

Don't merge this PR, since it contains a change that'll break users code.

This PR contains 2 commits. The first one is the a change to match esp-idf convention for idf.py target names.

In esp-idf, all targets you can run use dash for word separation, but frogfs uses underscore for the generate_frogfs_bin target (but not frogfs-flash). So the first commit rename the target to fit esp-idf's conventions (this was spotted by our CI tooling).

However, this is a breaking change for any user of frogfs (although, if they use frogfs-flash, it still works, so I would say it breaks developers of frogfs more).

The second commit is a dumb but useful one. If zopfli is present on the system, it'll use it instead of zlib for deflating the input files. zopfli is more efficient at compressing data while still generating deflate compatible stream (so it'll be decompressed perfectly by zlib or gzip or ...). For an embedded system, this is a simple way to pack more data in the filesystem's partition. If it's not present, it'll fallback to zlib.

I've modified the requirements.txt file to include zopfli so we can test it in our pipeline, but I'm not sure it's right (since it'll force all user to download zopfli). I don't know what you think here?

jkent commented 1 month ago

I think a way to install dependencies as needed is in order. I'd like to do the same for brotli too. Something similar to how node.js packages are installed on-demand. I toyed with the idea, but there isn't a stable api interface to pip. Maybe multiple requirements.txt files named similar to the transform?

X-Ryl669 commented 1 month ago

I don't know if a os.system('pip install brotli') could work in the same try: import() except: os.system('pip install brotli') import () way. Maybe there's a pythonic way to do so, I don't know.

X-Ryl669 commented 1 month ago

Maybe this module will solve the issue well.