(Extremely Niche) Feature Request: Detect Gzipped data in the output buffer

[EmperorArthur]

Okay, now this is a strange one and the use case is probably so niche it's not worth worrying about. However, I have permission to share the code, and this might be useful, at least as an application note or example.

Background

A neat trick I have found is to gzip data before storing it, then rely on the browser itself to decode the data on the other end. This works since, as long as the right header is set, all modern browsers handle it. Which is how I am able to serve an entire 150kiB SPA from a microcontroller without any additional storage.

How to generate an embed of any binary file:

gzip --keep <file_name>
xxd -i <file_name>.gz > embeds.h
Use a regex to adjust xxd's output to "const unsigned char <file_name>[] __attribute__((section(\".fin9\"))) = ..."

Code currently used:

bool is_gzip(const uint_farptr_t data, size_t length) {
    if (length < 2) {
        return false;
    }
    byte header[2] = {pgm_read_byte_far( data + 0), pgm_read_byte_far( data + 1)};
    if (header[0] == 0x1F && header[1] == 0x8B) {
        return true;
    }
    return false;
}

if (is_gzip_P(raw_bytes, length)) {
  res.set("Content-Encoding", "gzip");
}
...

Proposal

Adjust Response::setDefaults to add a check if the data to be sent is gzipped or not. If it is, set the appropriate header.

As I said, this is certainly something that does not have to live in the server, but it allows for some crazy things.

[EmperorArthur]

Actually, given what I am allowed to share is rather broad, what we do is add extra_scripts = pre:generate_embeds.py and have it run the included Python script.

At the least you might be able to turn this into a neat example.

generate_embeds.zip

[lasselukkari]

Have you seen the awot-scripts? It has built in tooling for creating those gzipped payloads with proper headers. The payloads can be also stored on a sd card if the flash is not enough. See the CardFiles.ino. Also see the tutorial here: Arduino Full Stack Tutorial: Deploying a React app on the ESP32. Here is an example of the output the script generated: StaticFiles.h. Also see DuinoDCX.

[EmperorArthur]

I have not, and will have to look into it. Thank you for pointing that out. I suspect that this will integrate extremely nicely in my Laravel mix project.

Largest issues that the code above addresses are AVR far pointers and automatic chunking. Both to deal with the whole 16 bit pointer/AVR paging and int16_maximum storage size. While I haven't worked with the ESP32, I suspect that code is unneeded on it since it appears to use 32 bit pointers.

[lasselukkari]

The script does the chunking too. I have tested it with arduino uno.