Resource framework / PROGMEM compression

[yatli]

@jmamma https://github.com/siara-cc/Unishox_Arduino_Progmem_lib

We can use this to develop a resource management framework like ".big" files for a game

[jmamma]

that's awesome. the decompressor uses up 3000bytes. how much do you think we'll save?

[yatli]

hard to tell, and perhaps it's more effective to concat shorter strings into a big chunk. I'll give this a try later.

[yatli]

Total compressable object size: 15489

[yatli]

The key property to exploit is that we don't need everything decompressed at all time. My idea is to allocate a pool to cache these values. During the setup routine of a page, call the resource manager to pull relevant stuff into the pool.

[yatli]

shox doesn't work for binary data. https://github.com/pfalcon/uzlib may be a better choice.

[jmamma]

cool. we'll find something that works. the principle is sound.

Comes at a good time, I can't enable debug mode anymore.

[yatli]

== Initial comp branch size ==
Sketch uses 228562 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 47314 bytes (83%) of dynamic memory, leaving 9517 bytes for local variables. Maximum is 56831 bytes.

== With uzlib ==
Sketch uses 231026 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 51610 bytes (90%) of dynamic memory, leaving 5221 bytes for local variables. Maximum is 56831 bytes.
PROGMEM +2464
RAM +4296 (4096 buffer)

== Compression run result ==
 ┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource                                                                                                                                                                                                  comp ◇◇ ┆ Yatao
 └─╜ ╰▷ .\gen-resource.ps1
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\fonts.cpp
compressed 3621 to 2871 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\icons.cpp
compressed 1986 to 1385 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\machine_names.cpp
compressed 5345 to 2947 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\menu_options.cpp
compressed 738 to 399 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\tuning.cpp
compressed 810 to 700 raw bytes

What do you think? Note, this framework is general so we can replace gzip with better compressors (as long as a lean decoder exists for arduino)

[yatli]

See #162

Current net gain is around 2KB.

[yatli]

https://github.com/google/brotli

[yatli]

Brotli destroys gzip:

-a----         4/11/2021   4:38 AM           2279 fonts.br
-a----         4/11/2021   4:10 AM           2871 fonts.z
-a----         4/11/2021   4:38 AM           1085 icons.br
-a----         4/11/2021   4:10 AM           1385 icons.z
-a----         4/11/2021   4:38 AM           1521 machine_names.br
-a----         4/11/2021   4:10 AM           2947 machine_names.z
-a----         4/11/2021   4:38 AM            314 menu_options.br
-a----         4/11/2021   4:10 AM            399 menu_options.z
-a----         4/11/2021   4:38 AM            501 tuning.br
-a----         4/11/2021   4:10 AM            700 tuning.z

 ┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource                                                                                                                                                                                                  comp ◇◇ ┆ Yatao
 └─╜ ╰▷ gci *.hex | Measure-Object -sum -Property Length

Count    : 5
Average  :
Sum      : 12500
Minimum  :
Property : Length

 ┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource                                                                                                                                                                                                  comp ◇◇ ┆ Yatao
 └─╜ ╰▷ gci *.z | Measure-Object -sum -Property Length

Count    : 5
Average  :
Sum      : 8302
Maximum  :
Minimum  :
Property : Length

 ┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource                                                                                                                                                                                                  comp ◇◇ ┆ Yatao
 └─╜ ╰▷ gci *.br | Measure-Object -sum -Property Length

Count    : 5
Average  :
Sum      : 5700
Maximum  :
Minimum  :
Property : Length

😱😱😱

[yatli]

net gain unknown, need to port brotli to avr

[yatli]

Decomposing icons.cpp:

[yatli]

About 'Elektrothic' Can we trim this one down to digits only.

[yatli]

== With unpack ==
Sketch uses 228944 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 47314 bytes (83%) of dynamic memory, leaving 9517 bytes for local variables. Maximum is 56831 bytes.
Low memory available, stability problems may occur.
PROGMEM +382
RAM +0 (hey)

Awfully low overhead.

[jmamma]

About 'Elektrothic' Can we trim this one down to digits only.

Yes. But we need to identify which bytes in the bitmap structure correspond to the numerals.

[jmamma]

comp branch