Pre-gzip (most of) the content

[mpvader]

Just like we recently figured out that how the 30MB large WASM file of the gui-v2 project is served is unnecessary slow, the html5 app is suffering from the same problem.

nginx is doing gzip compression on the fly, and that is hurting more than that it helps. See these screenshots, made by using a laptop in the Groningen wifi network, with the device being the CCGX of the Groningen ESS three phase system.

That CCGX, which is one of our more slower models, is typically at 20 to 30% idle CPU (only).

Zipping the 500kB file takes 4 to 5 seconds.

Just disabling gzip already speeds up the whole app by a lot.

Time needed for gzipping the largest file from the commandline

root@ccgx:/var/www/venus/app/static/js# time gzip -c ./90.4cdb7ffb.js > /data/home/test.gz

real    0m5.647s
user    0m1.489s
sys 0m0.209s

Load time with default nginx config (gzip is on)

Load time with gzip disabled:

[mpvader]

@frankwaalkens : am I correct in thinking that adding .gz files to the asset is something that is best/easily done as part of the CI steps in this repo?

If not, we can add creating those to the Venus OS build system.

Please make sure to check with Dirk-Jan, to see if the nginx config also needs adjusting.

[mpvader]

@frankwaalkens : pls do check back with me before you work on this, it might be that we'll make some Venus OS generic solution for this anyway - to prevent having to solve this in each and every web app itself.

[frankwaalkens]

@mpvader I think that is correct yes. How I see it this could be perfectly done in the build pipeline (adjusting the YAML file in frontend project). We can add a new step between the frontend build step and the step that puts the output inside the zip. We could experiment with these frontend output files first, they are relatively large:

[martinbosma]

@mpvader what is the status of this? Should @frankwaalkens look into this or is this already/going to be taken care of in the venus recipes?

[wiebeytec]

Webpack (used in the project), has the ability to make gz files of static content. We've set the threshold, somewhat arbitrarily looking at the generated files, at 500kbyte. There's only one candidate file really.

Venus' nginx config needs adjusting. I'll take care of that a branch in meta-victronenergy.

The gains are moderate on a Cerbo. Full request time (which includes remote decompression) for a 1.4 MB javascript file was 115 ms with dynamic gzip. Pre-gzipped was 70 ms.

Interestingly though, on the command line it takes 700 ms:

# lh
-rwxr-xr-x    1 root     root        1.4M Feb 13 14:05 90.3642f09d.js

# time gzip < 90.3642f09d.js > /dev/null

real    0m0.695s
user    0m0.678s
sys     0m0.011s

On a CCGX it's also not as bad as 4 seconds. When you do a test that doesn't use the nand/storage:

# ls -lh 90.3642f09d.js
-rwxr-xr-x    1 root     root        1.4M Feb  8 22:41 90.3642f09d.js

# time gzip < 90.3642f09d.js > /dev/null

real    0m1.628s
user    0m1.418s
sys     0m0.063s

Still, it's easy to do and every bits helps.

[wiebeytec]

@frankwaalkens : am I correct in thinking that adding .gz files to the asset is something that is best/easily done as part of the CI steps in this repo?

If not, we can add creating those to the Venus OS build system.

Please make sure to check with Dirk-Jan, to see if the nginx config also needs adjusting.

The branch I prepared is only for /app. The risk of enabling it globally is that there is a theoretical risk of the gzip file not matching the original (get stale for instance). With the GUIv2 and this HTML5 app it's part of the build process and correct.

Currently there are no other *.gz files present in /var/www/venus. I think it should be safe for us to enable globally?

[fwaalkens]

Solved in frontend: https://github.com/victronenergy/venus-html5-app/commit/793fc13294fac534fbaf755f4a815bf7e4f43305