search

letscontrolit / ESPEasy

Easy MultiSensor device based on ESP8266/ESP32
http://www.espeasy.com
Other
3.28k stars 2.22k forks source link

new UI #2314

Open ppisljar opened 5 years ago

ppisljar commented 5 years ago

lets create new UI based on modern web technologies

Benefits

i am developing basic version of this in a separate repository here: https://github.com/ppisljar/espeasy_new_ui

ESPEasy Checklist

json endpoints

cleanup

other

TD-er commented 5 years ago

And just to add to this issue/to-do list; This has been discussed for a while in the #dev channel on Slack.

ppisljar commented 5 years ago

for now we will add a download button to the tools page, which will only show up if there is enough space on spiffs. This means it won't be available on 1MB versions.

we are looking into either:

micropet commented 5 years ago

I think the new UI initially very confusing. With the old UI, you could immediately see the status of the sensors on the device side.

How can I switch off the UI?

clumsy-stefan commented 5 years ago

More as a remark/documentation...

It seems that the preparations for the new UI together with the new JSON endpoints increase the binary size significantly. Arduino IDE compiles for the 1M modules are now again about 10k over the 603k limit for OTA flashes... not sure about the platformIO generated binaries though.

This is really more a feedback and if that increase is ok/intended there is no need to worry about it ;) Also I'm happy to look again in size reducing possibilites, but I'm away for the next 3 weeks...

ppisljar commented 5 years ago

@micropet go to /filelist and delete the index.htm.gz file

micropet commented 5 years ago

Thank you

ppisljar commented 5 years ago

with latest version there is also tools -> back to old ui button

ppisljar commented 5 years ago

also, latest version shows live device values just like the old ui.

TD-er commented 5 years ago

About the size. We should remove the new stuff from the minimal builds. Eventually the main reason is to reduce the overall size of the build when the new UI is ready. This is a pre-alpha version, so be careful

clumsy-stefan commented 5 years ago

probably @ppisljar could embrace the changes in #ifdndef BUILD_MINIMAL_OTA's or better some new #ifdef BUILD_NEW_UI or so which can selectively be included or excluded for builds... at least as long as it's not completed and the old stuff is still in....

TD-er commented 5 years ago

The 'minimal' builds should not be included in the nightly builds if they will not fit, so no units will be broken until these defines are added, right?

uzi18 commented 5 years ago

@TD-er or if size is too big for OTA, rename it _no_OTA

clumsy-stefan commented 5 years ago

@TD-er probably platformIO is still able to build small enough binaries... ArduinoIDE is not.. but I agree, minimal builds should always fit in OTA...

what is also not so nice is, that the unit will flash a too big binary and then just not boot anymore (eg. crash on boot and loop)... that's what happened to mine ;)

uzi18 commented 5 years ago

@clumsy-stefan, maybe it is because image overwrite spiffs part of flash? eboot is really simple, does not check boundaries. Are you able to provide boot log in these conditions?

TD-er commented 5 years ago

@clumsy-stefan What image on what kind of node (flash size) is causing this? Then we need to add more checks to the build script to make sure that will not happen.

clumsy-stefan commented 5 years ago

@TD-er I used my self-compiled 1M image (which now unfortunately has 616KB when compiled with ArduinoIDE and I didn't realize) and the 2 step ESPEasyUploader provided in the dist of ESPEasy project... Normally this works... Probably the ESPEasy 2step uploader does not check for file size?

@uzi18 yes, I guess that is the problem... but again, unfortunately I only realised after flashing ;)

Fortunately the 1M units I mainly use for testing purposes and do not pan to use them currently... this may change though... also all other (40ish) 4M and 16M nodes updated withot issues ;)

TD-er commented 5 years ago

Yep the OTA flasher doesn't check for size, that's why I check for the free size before enabling the button and make sure no larger files will be entering the ZIP file.

clumsy-stefan commented 5 years ago

hmm... but that works only for the ESPEasyUploader (eg when running ESPEasy itself) but then you anyway only load the Uploader... that one should then check for the size... so ESPEasy should check for max. uploader size (eg. MAX-actual bin) and the Uploader for the ESPEasy size (eg. MAX-uploader bin)... or am I wrong here?

TD-er commented 5 years ago

Maybe we can split the OTA upload size into another issue? It is unrelated to the new UI, since it was always an issue and I'm not entirely sure we know the size at the moment the upload starts, nor if we can stop/prevent it when it is too big. Also the small 2-step OTA image doesn't check for anything and ESPeasy only handles the small 2-step OTA image.

clumsy-stefan commented 5 years ago

@TD-er completely agree! it just came up here because the binary became too big after including the new UI preparations...

ppisljar commented 5 years ago

@TD-er can you reference the OTA uplaod size issue here once its created. Browser will send the content-length header, so we can check that one for how big file it wants to send us, and we can respond with 500 before we store any byte to flash if we realize its not gonna fit.

TD-er commented 5 years ago

@ppisljar The problem with OTA is mainly for nodes with 1 MB. Then you have to perform a 2-step OTA, which means you first have to flash a very small image which can then be used to OTA the new one. So the running ESPeasy version will never see the "too big" image, but only the smaller OTA image. Thus in order to prevent these issues, we must patch the small OTA image and if I ever recompile that one it is very likely the image will become larger.

ppisljar commented 5 years ago

what about solving it on the client side ? we have the sysinfo json ... we know how much space your image can take and warn you if its too big before you even start the upload.

TD-er commented 5 years ago

Well then the user must first provide the new image, which will not fit and then use the 2-step OTA image. It still demands an extra step for the user to do. N.B. The image flashed first is not the one that's causing the issue here.

uzi18 commented 5 years ago

@ppisljar @TD-er with this idea it could be possible to integrate 2 step ota update, first fetch from internet and maybe also selector for official releases/update just not. reload whole. page and wait for connection, maybe it is needed to update 1st step. image to be detectable... with /sysinfo.json

clumsy-stefan commented 5 years ago

my opinion: the ESPEasyUploader bin should just have an additional check for the uploaded binary size and, if it exceeds, just not flash it. Same with ESPEasy itself, it should check if the new bin fits (no matter if it's a new ESPEasy or the 2step. uploader) and stop after max size (or not even start if the content-legth is too big).

besides this, IMHO new feature like this cool new UI should be made optional via #define until it's finished and the other one is kicked out (er eben switch between the two and only have one or the other enabled, to save bin size)....

TD-er commented 5 years ago

@clumsy-stefan As you may not have noticed, but apart from the extra space needed by the JSON bits, the new UI part does not take any space on the binary part of the image.

And recompiling the minimal OTA image is something which may give other issues. For example it is very unlikely it will get smaller (we need to add new features and the toolchain has changed) and then we have two of these OTA images. Which one should we then use to compute the max. allowed OTA-upgradable image?

@uzi18 One of the new ideas with this approach of the new UI is that you can indeed add a more dynamic upgrade path and even upgrade the UI separately or switch between versions. (e.g. simple version or elaborate version)

uzi18 commented 5 years ago

it is just to rebuil ota helper, smaller=better, we can compute max allowed size of image to be sure it will not overwite spiffs during 2 step ota upgrade

@TD-er and stay tuned for next expressive sdk it will free more memory, if Arduino guys will use it :)

clumsy-stefan commented 5 years ago

@TD-er about the UI, agree, it just seems the json parts do take quite some space...

about the OTA process: probably I really do get something completely wrong, but I thought it should be possible at run-time to calculate the max. free size for a new binary ("max available size in total" - "current running binary"). No matter if it's an ESPEasy binary, a uploader binary or whatever... it should always the actual binary be that decides how mus space is left for a new one... it can also be that after flashing the uploader the next binary is not an ESPEasy, or another uploader etc...

eg. if the ESPEasy has 650k and therefore the flasher (and another of the same binary) would have no space, because we "assume" the new binary has the same size (or bigger) or the flashe has a specific size we shouldn't generally disallow updates, beacuase probably someone finds a way for a new, much smaller uploader, which then would not be able to uploaded via OTA, because of a "wrong" size assumption...

isn't that calulation of "free space left" possible to do at run-time?

TD-er commented 5 years ago

isn't that calulation of "free space left" possible to do at run-time?

It is and I already do it in ESPeasy. See the code surrounding the OTA button on the tools tab. But my point is that rebuildling the 2-step-OTA firmware will almost certainly increase its size and thus reducing the max. size of ESPeasy firmware. Things that will increase that small firmware:

The last part is what worries me the most, since I've been trying to rebuild our own firmware with core 2.3.0 a few months ago and didn't succeed even on the minimal OTA firmware to make it fit (iRAM size didn't fit) So I have no idea what the current tool chain for building an image has changed and thus may do to the binary size. And we need to have some kind of ESPeasy-based firmware for this, to be able to read the WiFi credentials. There are all kinds of workarounds possible, but they will also limit the use of that small OTA firmware to only newer builds. (e.g. temporary duplicate the wifi credentials in the "EEPROM" reserved section of the flash)

uzi18 commented 5 years ago

@TD-er check this: https://github.com/espressif/ESP8266_NONOS_SDK/issues/162

clumsy-stefan commented 5 years ago

@TD-er agree with all exept that you're testing for the new ESPBinary size and compare to fixed values and then decide to show the button or not and not to what the client wants to send, if I understand the code corectly?

uzi18 commented 5 years ago

assume 90% people use ota update and serial flashing only when they are forced to (1st time/something went wrong/testing) .

TD-er commented 5 years ago

@clumsy-stefan The button is shown when there is room for the small OTA image. So it is indeed checking against the current installed firmware.

The current work flow does not allow to check for the provided image size and also ESPeasy will not be handed the final binary. So we're stuck here until there is some way to keep track of the entire process of the update. The new UI can help here, since the UI can remain active in the browser and doesn't need the ESP to do the checks. It can also remain active while the small OTA firmware is running.

TD-er commented 5 years ago

Should I disable only the newly added JSON parts for the BUILD_MINIMAL_OTA or all JSON parts in the web server? N.B. I will use a define for WEBSERVER_NEW_UI to include these newly added functions.

clumsy-stefan commented 5 years ago

Hmm.. difficult to say, why not adding a general '#define USE_JSON' which includes all JSON that are not used by the UI and a '#define USE_NEW_UI' for all JSON and non-JSON parts that the new UI uses? Would probably be the most flexible approach?

TD-er commented 5 years ago

For now, I just moved all the new-UI related JSON parts to be wrapped in this WEBSERVER_NEW_UI and also added some more to remove the extensive sysvar page and timingstats page from the minimal OTA version. This makes the minimal OTA build somewhere around 585kByte, so it should fit again.

clumsy-stefan commented 5 years ago

@TD-er great, thx! unfortunately I can't really test currently, as I'm abotu 12k km away and only have limited internet access ;) but probably I can try to compile here and compare file sizes...

TD-er commented 5 years ago

Well, those smaller builds are just perfect for the "limited internet access" :) But just make sure to test it locally first, since there have been some issues with the build tool chain the last few days. So when the node keeps rebooting, throw away the platformio framework cache directories and build again. (and perform a pio update)

clumsy-stefan commented 5 years ago

😁 I only have a 16M unti with me (just for fun) and I won't flash anything at home while beeing that far away with no chance to stop it from looping or so.... even typing comments here from time to time on the mobile is really challenging already 😁 but at least I have the build-environment here (Arduino IDE), ~so I'll try later on to compile...~

EDIT: Arduino IDE compiles it now to 597k, which should fit again! Thanks!!

clumsy-stefan commented 5 years ago

@TD-er

The new UI can help here, since the UI can remain active in the browser and doesn't need the ESP to do the checks. It can also remain active while the small OTA firmware is running.

This presumes that the updates are done via browser... but for bigger installations that won't be true, mostly the updates then would be done by a script calling /update on (all) units and doesn't handle any javascript or similar... so making sure the Uploader does not accept more data than available space or at least don't write more, would anyway be neccesary... IMHO...

TD-er commented 5 years ago

@clumsy-stefan If an update is done like that, the update script could also do these checks, or the pre-deployment check would also indicate an issue I guess ;)

clumsy-stefan commented 5 years ago

@TD-er agree... Actually I never thought of that... I guess I'll include this in my deployment scripts... However this solved it 'only' for me....

TD-er commented 5 years ago

You may get some inspiration from the before_deploy script in the ESPeasy branch. In there I also check for the size of the binary to see if we will include it in the ZIP file.

Grovkillen commented 4 years ago

For the sake of documentation, I have a portfolio here http://grovkillen.dribbble.com/ , that shows progress of the new GUI.

Grovkillen commented 4 years ago

https://github.com/letscontrolit/GUIEasy