Hosting webpage from Flash memory instead of SD card

[karoria]

This is just a request if we can host http server (index.html.gz) from Flash memory instead of SD card. For Teensy4.1, we have very large amount of flash memory unutilized and I think we should use it unless it has negative implications for hosting webserver. Further, there are chances to accidently delete "www" directory from SD.

[terjeio]

Making some read-only files available from flash can be done in the plugin code by converting them to arrays and adding filing system primitives so that the http daemon can read them, I guess that should be made an option. A downside to this is that updates requires a rebuild/reflash.

I believe adding a generic writable flash based fileystem will be a lot more work, so I'll leave that for somebody else to tackle.

[karoria]

OK. Got it. But I see rebuilding/reflashing for http server as "positive" point as this is a one time procedure only just like we flash grblHAL, the webui should also part of flashing. This will make webui files read only and hence they will be safe. I like the idea to make it an option in code.

[terjeio]

Option to add read-only files to flash added in build 20211010.

I have written a windows-only command utility to convert files to the required c structure, bin2httpd:

bin2httpd.zip

It is written in C# and is pretty primitive (no error handling), syntax is bin2httpd <filename> where <filename> is path to the file to convert. It generates a file with the same name but with a .h extension.

Here is how files can be incorporated in code, the last few lines are important! The files has to be registered with the virtual file system as well, that is done in the webui server code:

https://github.com/grblHAL/Plugin_WebUI/blob/878d6e08334f75ad4a4202314c79138107d061fe/server.c#L953-L955

[karoria]

That's amazing. Thanks for your efforts. I will definitely experiment with that.

[karoria]

Hi @terjeio After the recent update regarding littlefs and webui server code, I am still not able to run a webui from ESP3D interface. I am using Teensy4.1 board. The basic webpage to upload webui files shows up. I upload index.html.gz from https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/dist/CNC/GRBLHal/index.html.gz. But after upload it does nothing. I also want to know how I can upload my grblTouch webui in littlefs which comprises of loose files index.html, style.css, app.js and other support files (All these loose files takes hardly 200kb in total). I couldn't find a tool to compress all those files to one index.html.gz in case it is compulsory to do so. Please help.

[terjeio]

But after upload it does nothing.

No errors in the browser? Check network traffic by open the developer tools by F12 and reload. No request for the root document? Or waiting forever? Here is what I get in Firefox:

Did you have use the ?forcefallback=yes argument to open the maintenance page? If so you may have an index.html or index.html.gz file on the SD card that takes precedence.

I couldn't find a tool to compress all those files to one index.html.gz in case it is compulsory to do so. Please help.

Try with compressing the index.html separately and refer to the other files with absolute paths starting with /littlefs/, this might work - but I have not tested it.

Alternatively you have to write your own server.c implementation and add redirection to littlefs for the files you want to load from there. Note that redirection function will only be called if the file in question is not found on the SD card.

[karoria]

No errors in the browser? Check network traffic by open the developer tools by F12 and reload. No request for the root document? Or waiting forever?

Chrome developer tools shows "Failed to load resource: net::ERR_CONTENT_DECODING_FAILED" after uploading index.html.gz and refreshing the url. Here is screenshot.

Did you have use the ?forcefallback=yes argument to open the maintenance page? If so you may have an index.html or index.html.gz file on the SD card that takes precedence.

After using ?forcefallback=yes argument, it defaults to index.html from SD card. I noticed here it doesn't follow path www/index.html instead it opens index.html from root of SD card.

Try with compressing the index.html separately and refer to the other files with absolute paths starting with /littlefs/, this might work - but I have not tested it. Alternatively you have to write your own server.c implementation and add redirection to littlefs for the files you want to load from there. Note that redirection function will only be called if the file in question is not found on the SD card.

I experimented with uploading webui files to root of littlefs after changing paths and it worked really well. However, I don't understand why it occupies more space (almost 3 times) as shown by progress bar on the interface. It shows total memory 512KB and occupied was 128KB before copying any files. But as I copied some files, it shows 100% occupation despite my files were approx only 60KB in total. I can confirm that all files integrates and functions well this way but I don't get enough storage space due to this issue. Screenshot is attached for your reference.

[terjeio]

Chrome developer tools shows "Failed to load resource: net::ERR_CONTENT_DECODING_FAILED"

Then the file has been corrupted somehow - does it work if you place it in the root of the SD card?

I noticed here it doesn't follow path www/index.html instead it opens index.html from root of SD card.

The search sequnece is the root of the SD card first if the path is relative, if this fails then the redirect function is called.

But as I copied some files, it shows 100% occupation despite my files were approx only 60KB in total.

How much space the files occupies is related to the sector size in the flash which is 64K for the Teensy 4.1, each file needs a whole number of sectors. In addition sector(s) are needed for the root directory and redundancy/failsafe handling.

A side note: the STM32F7xx driver cannot use littlefs due to the flash sector size beeing 128K and the flash beeing quite small.

I don't get enough storage space due to this issue.

It can be expanded - I set the size fairly low since the WebUI file is small. It can be expanded here, change FS_SIZE.

[karoria]

Then the file has been corrupted somehow - does it work if you place it in the root of the SD card?

It seems I can't download the file from the said GitHub repo. I have tried safari and chrome browsers but no luck. Will you verify that the index.html.gz file at https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/dist/CNC/GRBLHal/index.html.gz works with webui?

How much space the files occupies is related to the sector size in the flash which is 64K for the Teensy 4.1, each file needs a whole number of sectors. In addition sector(s) are needed for the root directory and redundancy/failsafe handling.

I got it. This means a single file with say 200kB occupies less space than in my case multiple files with less memory.

It can be expanded - I set the size fairly low since the WebUI file is small. It can be expanded here, change FS_SIZE.

I changed FS_SIZE to 4096 instead of 512 for my Teensy board and my storage issue is over. I can keep all loose files of grblTouch in uncompressed form and the UI works well. So at present, I am happy without file compression. I have tried 2-3 ways of compression using tar.gz and then renaming it to html.gz but the file integrity is lost and raw data of file is shown instead of web app mostly may be due to index.html doesn't get priority there. So, right now I am leaving it uncompressed as I don't have storage issue for now.

Some more points:

1. I have to type /littlefs/index.html to get my UI instead of only. What should I change to do it right?
2. WEBUI_MAX_CLIENTS is set to 4 by default but it still doesn't seem to connect to more than one client at a time.

[terjeio]

The index.html.gz works for me.

I have tried 2-3 ways of compression using tar.gz

Can you attach the compressed file to a comment, you may have to zip it first. BTW - have you tried 7z for compressing to gz?

I have to type /littlefs/index.html to get my UI instead of only. What should I change to do it right?

Somehow manage to compress index.html to .gz successfully.

WEBUI_MAX_CLIENTS is set to 4 by default but it still doesn't seem to connect to more than one client at a time.

Only one client can connect at a time and claim control (there is only one set of "serial" buffers). WebUI has a switch function implemented at websocket level where the latest to take control takes over and the previous is notified about losing it.

If you want to have more than one active client in control you have to figure out how to handle that and add or update code accordingly. It may be possible by adding event handlers to the websocket events, but I am not sure...