Closed luc-github closed 1 year ago
An initial update for WebUI 3.0 of the grblHAL WebUI plugin is now available. I am testing with a Nucleo STM32F756 board and the plugin should also work with the iMXRT1062 (Teensy 4.x) driver. Note that the ESP32 grblHAL driver does not use the plugin, I will backport the code changes to that later on.
@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.
Ok understood - unfortunatly I have no STM boards, neither Teensy 4.x, but I can emulate webserver and board responses in webUI dev mode
I saw the the https://github.com/grblHAL/Plugin_WebUI but I was thinking was same code in https://github.com/grblHAL/ESP32 so I did not realized it was V3 based - I will have a look
@terjeio you wrote
my plan to contact you about some issues I have encountered
Can you tell which ones ? I can start by these first
I have added the code base to test and build grblHAL webUI https://github.com/luc-github/ESP3D-WEBUI/commit/bac40f99ee64e1ed310ba924acc120e5f1bcf166
I need some confirmation :
Is grblHal
typo correct ? I saw also different writing like GRBLHal
and GRBLHAL
do you have a logo for grblHal ? I have only found
is there and svg version of it ?
about github link Do you want to go to https://github.com/grblHAL/ ? or target the specific FW like https://github.com/grblHAL/ESP32 or https://github.com/grblHAL/STM32F7xx, in that case it would need extra info in [ESP800] response
about webUI update I see on esp32 version you cannot update the webUI as do not use the maintenance page and directly flashed the webUI, is that the same with pluggin ?
I will check what play SD job commands you are using, reading https://github.com/grblHAL/core/wiki/SD-Card-implementation,-a-technical-walktrough it seems $F=<filename
but I will double check
Issues encountered:
ESP701 for querying the status of a running job suddenly (randomly) is no longer sent after a running a few. Reloading resumes the querying.
If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing.
The default file system is FLASHFS, could this be the first listed in the target supportedFileSystems
(in files.js) or perhaps read from the ESP800 response?
I've changed the code above and it seems to work but not sure if this has any side-effects (other than failure if none are specified):
if (useUiContextFn.getValue("autoload") && currentFS == "") {
currentFS = files.supported[0].value
onSelectFS()
}
Is grblHal typo correct ? I saw also different writing like GRBLHal and GRBLHAL
"grblHAL" is the correct version. The welcome message from the controller is not correct, but I have been reluctant to change it as it may break senders.
do you have a logo for grblHal
There are a few proposals but none has been decided upon. I have to make my mind up...
Do you want to go to https://github.com/grblHAL/ ?
Yes, I have changed this already.
I see on esp32 version you cannot update the webUI as do not use the maintenance page and directly flashed the webUI, is that the same with pluggin ?
Yes. For the plugin changing this will require a flash filing system. I may add this later. If the WebUI is compiled into the binary then adding index.html.gz to the SD card www folder will override the one from the binary so there is an update path available. More work on this is needed on my end - perhaps adding download of a new version to the www folder as a first step?
I will check what play SD job commands you are using
I am using $F=, I can also use ESP220 (IIRC this is used by WebUI 2.1) but when I check your ESP3D source it seems to me that this (and some other commands) has been repurposed? The commands that may have been reporposed have the help text "??" in this list.
I see extra real time commands: ... Is that list complete / up to date ?
I am not sure it is up to date, I'll have to check. Most can be ignored? Perhaps the most important change is the new Tool state, it would be great if WebUI could support grblHAL style tool change handling. And yes, this works even when running a job from the SD card.
Here are the preferences and target folder I created:
Should grblHAL get its own firmware target name and id?
Some ideas:
Can WebUI fetch the settings labels from the controller? grblHAL has dynamic settings handling - plugins may add new ones, even user defined plugins. In my sender the settings UI is built from information requested from the controller. I can easily add an ESP command for fetching a json formatted list of setting ids and labels (and even descriptions) in a json formatted response.
On a longer term todo list will it be possible to use the WebUI style settings handling? It will require the support for floats and a flag type where bit 0 controls the enable status for the rest.
ESP701 for querying the status of a running job
Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel
If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing. I need to check but I may did that on purpose to display or not the error message
The default file system is FLASHFS,
This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet
I am using $F=, I can also use ESP220
Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0
I am not sure it is up to date, I'll have to check. Most can be ignored
it is just to allow these command to be used by macro using a text descriptor as they are not printable,
I also see some commands : ?
,~
,!` need to be update by the not printable code to be handled properly in grblHal, so will do
would be great if WebUI could support grblHAL style tool change handling.
You will have to tell me more what you expect for UI and behavior, but I see no issue to support it
The current webUI use grblhal
as name, for the ID I did not defined it yet, but because it is not selectable for ESP3D, it is not major issue,
here the current list : https://github.com/luc-github/ESP3D/blob/3.0/esp3d/src/include/defines.h#L28-L36
If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW
Can WebUI fetch the settings labels from the controller? currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language
So actually you can add any sections / items you want in settings just follow current syntax, if need more type of settings, just need to define type, type constraint, rendering form and should be ok like others types
Reading your version now
I have updated and build the index.html.gz according your changes https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/dist/CNC/GRBLHal/index.html.gz
I still need to update the real commands parts
I have also noticed you do not support the one session feature
so many webUI can be open at once, is that a choice ? I won't recommend it
ESP701 for querying the status of a running job
Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel
Polling for the real time status continues, only the ESP701 request stops. If I remove the ESP701 request the stream status is not displayed even if it is available in the real-time report. I assume this could be changed to use the real time report instead?
The default file system is FLASHFS,
This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet
It is not present in the controllers using the plugin.
I am using $F=, I can also use ESP220
Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0
Ok, this explains it. They should not have been reassigned to a different use then? Or was this a customization made by bdring and his team?
If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW
If not needed I am ok with using ID=10 as I do not know what it is used for in your code.
Can WebUI fetch the settings labels from the controller?
currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language
Ok, what is the ESP command to use for returning those. And to be clear I am talking about the grblHAL settings where those not known by the WebUI does not have useful labels:
preferences content has some changes already, polling commands can have difference frequency
Ok, will update. Sun came out today after a period of rain - so it may take some time. And I will be away for a period starting Tuesday 2nd.
I have also noticed you do not support the one session feature so many webUI can be open at once, is that a choice ?
Definitely not - or at least until an abitration protocol can be added. Only one client should be in control at the time. As a first step I might add support for additional read-only clients - but this is longer term.
It is not present in the controllers using the plugin.
I need to check if I have added a dependency on ESP800 settings
[ESP800]
{
"cmd": "800",
"status": "ok",
"data": {
"FWVersion": "3.0.0.a111",
"FWTarget": "grblhal",
"FWTargetID": "80",
"Setup": "Enabled",
"SDConnection": "direct",
"SerialProtocol": "raw",
"Authentication": "Disabled",
"WebCommunication": "Synchronous",
"WebSocketIP": "localhost",
"WebSocketPort": "81",
"Hostname": "esp3d",
"WiFiMode": "STA",
"WebUpdate": "Enabled",
"FileSystem": "LittleFS",
"Time": "none",
}
}
grblHAL settings
these are in language pack - they are just translation from commands id : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/translations/en.json
the one session feature
this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want
I need to setup an esp32 system for grblHal as I am afraid I will do some blind code and may misunderstood some commands
if I replace the plugin files to existing ESP32 code would it work ? any other changes should I do - I can do changes manually and this will let me know also why current version is always in reset error on my board
Or was this a customization made by bdring and his team
In FluidNC they renamed all commands https://github.com/bdring/FluidNC/wiki/FluidNC-Commands-and-Settings
@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.
Just fyi, have started looking at this. First issue is that I used the STMCubeIDE provided middleware for SD card access (using DMA & SDMMC peripheral) - which uses an older FatFs version. Am looking into what it's going to take to upgrade..
@terjeio should I aim for R0.13c as per F4/F7 etc, or latest R0.14b?
the one session feature
this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want
Wait... One step at a time please. I have some support issues to clear before I can dig into the details.
if I replace the plugin files to existing ESP32 code would it work ?
No, it is a major refactoring. Ideally I should update the ESP32 driver and the plugin so it could be used directly - I do not like to maintain duplicate code.
why current version is always in reset error on my board
Oops, I forgot to asnwer this, by default grblHAL uses normally closed switches for all inputs. Connect the reset pin to ground and you should be fine.
@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.
@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.
Thanks, was actually the webui plugin that complained, may just be something minor...
WebUI is updated to a7
FileSystem
is set to none
[ESP800]
{
"cmd": "800",
"status": "ok",
"data": {
"FWVersion": "3.0.0.a111",
"FWTarget": "grblhal",
"FWTargetID": "80",
"Setup": "Enabled",
"SDConnection": "direct",
"SerialProtocol": "raw",
"Authentication": "Disabled",
"WebCommunication": "Synchronous",
"WebSocketIP": "localhost",
"WebSocketPort": "81",
"Hostname": "esp3d",
"WiFiMode": "STA",
"WebUpdate": "Enabled",
"FileSystem": "none",
"Time": "none",
}
}
Now will check : https://github.com/grblHAL/core/wiki/Report-extensions because additional grblHal states are not supported by webUI currently
about : Can WebUI fetch the settings labels from the controller?
it is possible just need to define the format you want to use and I can customize the grblHal panel for settings because it is specific, currently I use the GRBL $$
display but we can imagine another command that do same and add label and constraint for each like for ESP3D settings even section if necessary and it will dynamically build the UI accordingly
I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin. Add preferences.json to the root of the SD card.
I left the SerialProtocol
value to Socket
in the ESP800 response - not sure ift that is correct but it works for me.
There is flash filesystem support for the ESP32, I will change the Filesystem
value later to reflect that.
Note that forms based autentication is not yet supported for the ESP32 driver. I'll put improved settings handling down the list until the basic stuff is completed - forms based login and unified SD card handling are the largest items.
And BTW the ESP32 webserver is really slow compared to e.g. the iMXRT1062 controller where WebUI loads in the blink of an eye...
I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin.
Ok thank you I give a try tomorrow - I was out all day today and need to finish some code first
SerialProtocol value to Socket in the [ESP800 response] - it is fine - there is no use currently in WebUI but better to be consistent, if the need popup
Ho yes ESP32 sync webserver is very slow, the test I did on current grblHal is already far faster (it use the idf async http server right ?) I am not surprised on another chip the performances are better - on reprap they claim to have wifi file transfer of 1.2MB/s but they do not use esp webserver ... I have also noticed performances issues related to SD https://github.com/luc-github/ESP3D/issues/576, so I use SDfat instead of ESP32 SD library in ESP3D
thank you for the porting It will be easier for me to experiment - I will feedback you tomorrow
Hi I have flashed my board with latest git and webui 3.0, very cool job thank you,
so I have some questions :
Is that a choice to not display the IP at start ? it is painful when board is in STA/Client mode but start in reset and so it is locked , cannot send any command to know the IP, I see mdns is present but no SSDP, so on windows, the only choice is to find info on DHCP server ...
ESP800
[ESP800]
FW version:1.1f
FW target:grblHAL
FW ID:80
SD connection:direct
Serial protocol:Socket
Authentication:Disabled
Web Communication:Synchronous
Web Socket IP:192.168.2.108
Web Socket Port:81
Hostname:Grbl
WiFi mode:AP
File system:directsd
Hostname:Grbl
Some comments :
In STA mode - it still display WiFi mode:AP
File system:directsd
directsd
is not supported should be none
or spiffs
or littlefs
Above status lead me to add a new flag : hostFS
, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www, using the new flag hostFS would tell webUI where to load/save preferences.json, what do you think ?
Settings
the settings use old format of 2.1 instead of 3.0, so they cannot be loaded, and webUI raise this error :
WebUI and Firmware version are not compatible
, WebUI 3.X has new output format for all ESP commands, the same as you used in about page, I enclose both format
About page / [ESP420]
the CPU freq is
19
- I guess this is wrong 😄 or it is very slow setting
$G
give me [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0.]
what is the meaning of .
in S0.
?
Form another side, is that normal to have F0
instead of F0.0
as mentioned : https://github.com/gnea/grbl/wiki/Grbl-v1.1-Interface#grbl-response-messages
Because I need to know to adjust the regex catching patern to get informations and processing them, and there is not information on https://github.com/grblHAL/core/wiki/Report-extensions page
New flags / states That is for me to read https://github.com/grblHAL/core/wiki/Report-extensions and adjust each triger / rendering one by one
Sorry to ask so many questions, hope you can answer before you will going to vacations - so I won't bother you 😸
Edit I see: $ES
so I may use it instead of $$
to list grblHal settings, is that ok ? change command still seems same $<ID>=<value>
, right ?
Edit2: if I read well need to send $EG
then $ES
then $$
to be able to display settings properly ?
any reason not to add the value in $ES
? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of $ES
line format, sender knowing $ES command should not need the $$
, because $$
would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway
Is that a choice to not display the IP at start ?
Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.
This is the latest v3 json version:
ESP800
{"cmd":"800","status":"ok","data":{"FWVersion":"1.1f","FWTarget":"grblHAL","FWTargetID":"80","Setup":"Enabled","SDConnection":"direct","SerialProtocol":"Socket","Authentication":"Enabled","WebCommunication":"Synchronous","WebSocketIP":"10.0.0.22","WebSocketPort":"81","Hostname":"Grbl","WiFiMode":"STA","WebUpdate":"Disabled","FileSystem":"none","Time":"none"}}
Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested):
http://10.0.0.22/command?cmd=[ESP800]json=yes%20version=3.
the CPU freq is 19 - I guess this is wrong
Yep, it is the stepper code clock frequency, I have to add a new HAL value for the MCU clock...
$G ... what is the meaning of . in S0. ?
I guess it is for implying that the value is a float - inherited from legacy Grbl.
Form another side, is that normal to have F0 instead of F0.0
Good question, I have to check my code.
I see
$ES
so I may use it instead of$$
to list grblHal settings, is that ok ?
No, $ES
is for getting information about the settings, not the values. There is an internal API for querying setting details which I use to generate the ESP400 response, for most grblHAL settings this is relatively straight forward. The settings API is designed to allow user defined settings (in plugins or board support code), these are also possible to query and will show up in the $ES
response etc.
change command still seems same
$<ID>=<value>
, right ?
Correct. I try to be backwards compatible with the legacy Grbl specification, at least in spirit.
if I read well need to send
$EG
then $ES then$$
to be able to display settings properly ?
I would rather like to add a ESP command for querying setting details, same format as ESP400. I can even add most to ESP400 by just looping trough them instead of outputting those that your backend outputs.
any reason not to add the value in
$ES
? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of$ES
line format, sender knowing $ES command should not need the$$
, because$$
would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway
The reason is that $ES
is used once at startup (at least in my sender) to build the settings UI. $$
is then used to fetch the values as needed.
Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested):
Ok so I see the issue - I have manually sent [ESP800]
in terminal to verify settings when checking the File system:directsd
issue and that generate the problem
I would rather like to add a ESP command for querying setting details, same format as ESP400
The ESP400 management is not part of specific FW but common to all target ESP3D ESP3DLib GRBL_ESP32 grblHal etc...
if you want to merge all settings in ESP400 like format - I can remove the tab Features
for grblHal, and handle these settings in grblHal tab, which give you all flexibility for the additional types, I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX
for extended settings display
@terjeio What do you think ?
Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.
My question was because you already display
[MSG:WIFI STA ACTIVE]
so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO
When locked the ?
seems not working so I am not sure putting IP in report is necessary, additionnaly it will add extra info that should not change, and this info is already in $I
Final question : what sender currently support all/most grblHal extra features ? I have tried https://github.com/terjeio/ioSender/releases/download/2.0.40/ioSender.2.0.40.zip but it does not work on my Win11 and My norton AV does not like it, more over it remove each part of the sofware one by one saying there is a problem
This is mostly to have a reference to verify I interpret properly the new states in webUI
I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX for extended settings display @terjeio What do you think ?
IMO it will be better to add a new ESP command that outputs the information in json format, possibly the same as ESP400 uses as you already have parser/UI builder code for that.
My question was because you already display [MSG:WIFI STA ACTIVE] so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO
I can add that but there will still be a problem with some controllers that uses native USB for communication as most of these loses any info sent before a client is connected. Not easy this...
When locked the ? seems not working
It should work - are you sure about this.
Final question : what sender currently support all/most grblHal extra features ?
Only ioSender as far as I know.
... but it does not work on my Win11 and My norton AV does not like it
Is it Norton blocking it? I have not signed the code as the certificate for doing so is expensive.
This is mostly to have a reference to verify I interpret properly the new states in webUI
There is only one new state Tool
in the real-time report, Alarm
and Run
can have substates added if enabled in settings. These substates are added in the same way as for Hold
. The $G
report has some missing parser states added as well as some new ones.
IMO it will be better to add a new ESP command
if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings
It should work - are you sure about this.
I have reflashed to test the fixes in webUI I did and I cannot duplicate anymore ... I am stumped...
Is it Norton blocking it?
yes and deleting all files one by one, I have paused Norton and now everything seems working - only the SD tab seems unresponsive but it is working in webuI so - it is not a problem for me
The $G report has some missing parser states added as well as some new ones
Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 ___M5 M9 T0 F0 S0 _] forGRBL
Forget my comment about F0.0, it seems my GRBL fw do also
F0instead of
F0.0`
I will check everything - thank you for all feed back and enjoy your vacations, I have everything to work now I think - this will keep me busy for a couple of days ^_^
IMO it will be better to add a new ESP command
if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings
Yes - I guess this makes it simpler for both me and you. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask) and 6 (decimal or float if you will, later I might add double precision support in the core as an option...).
Examples in ioSender:
$22 - homing enable (exclusive bitfield) - can be emulated imperfecly by your B
type:
Disabled
Enabled
$5 - invert limit pins (axismask)
IMO this can be emulated by your
B
type so not really unsupported.
Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 ___M5 M9 T0 F0 S0 _] forGRBL
There are many more new ones, here is the source code in report.c and motion modes supported in gcode.h.
grblHAL also supports the G59.1
, G59.2
and G59.3
offsets.
If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller? Note that these can also be added to by third party code.
. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask)
ho yes mask controls are a pain - I had some before and I converted all them to simple booleans, but I will see what I can do
If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller?
No I am not - I will keep this part for then end because anyway all text need to be in translations packs so I currently just refer to code for that
There are many more new ones, here is the source code
Yes saw it - thank you
I have committed some updates, noatbly forms based login (as used by WebUI 3.0) should now work and I have added IP address to the WIFI STA ACTIVE message.
Ok will check that - I am still working on sanity check if not Flash is enabled - it was not a case I had planned so I am following every rat holes to be sure I did not forget anything
that bring me back to my question :
[ESP800] File system:"directsd"
directsd is not supported should be none
or spiffs
or littlefs
Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www,
So add a flag hostFS: "SD/www"
in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?
Note: I have prepared a code to handle Bit Masks, but I will wait your decision of the format of [ESP400] with all settings to implement corresponding controls
[ESP800] File system:"directsd" directsd is not supported should be none or spiffs or littlefs
none
is a bit counterintuitive, could virtual
be a more logical option?
Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www,
In fact it is possible to fetch the WebUI from flash, but then from a read only file system. I do check if the file is available in the www folder and if so it will used instead.
So add a flag hostFS: "SD/www" in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?
That could work, even when file system virtualization is implemented. Perhaps hostFiles
or hostPath
is a better name? Which file system becomes irrelevant when virtualization is implemented?
Just for the fun of I added a few lines to auto push the real time status report (along the $G-parser state report if enabled and changes to the report) and it works!
Just add this code before webui_init() if you want to test it:
static on_execute_realtime_ptr on_execute_realtime;
static void auto_report (sys_state_t state)
{
static uint32_t ms = 0;
if(hal.get_elapsed_ticks() >= ms) {
ms = hal.get_elapsed_ticks() + 3000; //ms
if(hal.stream.state.webui_connected)
protocol_enqueue_realtime_command(CMD_STATUS_REPORT);
}
on_execute_realtime(state);
}
and in webui_init() add:
on_execute_realtime = grbl.on_execute_realtime;
grbl.on_execute_realtime = auto_report;
Change $10 to enable auto-reporting of the parser state:
If this is of interest there should be a way to set the timeout?
File system:"directsd"
Well the wording is confusing now I should admit, initially it was meant for Flash only, as I did not expected Falsh was not available - I may rename it to FlashFS
instead for clarity
In fact it is possible to fetch the WebUI from flash, but then from a read only file system.
For this part it does not really matter where the original webUI is located as server already streamed it and it is loaded on client side, unless you want a read only preferences which make no sense I think - if FS is not accessible it is better to say there is no Flash FS available IMHO, it avoid user confusion and limit user interaction (and unexpected situation)
That could work, even when file system virtualization is implemented. Perhaps hostFiles or hostPath is a better name?
I think hostPath
is fine if on SD it would be SD/www
as done today, indeed only location is important, not FS actually
If this is of interest there should be a way to set the timeout? well if you put a time out configurable then you can use the timeout to disable the autoreporting using value of timeout to 0 and no need to put a flag in report option, just my 2 cents, another solution is no timeout but send report each time a command is received executed, or status change, but that may be more complex to implement
I will give a try tomorrow to this new version thank you 😸
Edit: I see your /
to serve files is on /
of SD so hostPath: www
should be ok, I must also change the path for SD files to allow download by removing SD/
Unfortunatly I cannot compile latest git I have raise issue https://github.com/grblHAL/ESP32/issues/40 I found some sanity check are missing but still have issues
I have also did some test on current version to test the upload of preferences.json
and found that any SD sub folder cannot be parsed or even have some weird behavior listing the root content sometimes,
Upload to these sub directories seems working as I got [MSG:Upload ok]
and I can download the file if I write proper path in browser, but listing is not working and this is not limited to www
subdirectory but all subdirectory
is that normal ?
EDIT:
I found the root cause of parsing issue of subfolders
command sent to grblHal :
http://192.168.2.108/sdfiles?path=%2Fconfig&action=list&t=1659321435917
give : {"files":[],"path":"/confi","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""}
which is wrong should be config
not confi
the last char seems trailed
confirmed by double it : http://192.168.2.108/sdfiles?path=%2Fconfigg&action=list&t=1659321435917
which give better output :
{"files":[{"name":"pen_position.json","shortname":"pen_position.json","datetime":"","size":"24 B"},{"name":"readme.txt","shortname":"readme.txt","datetime":"","size":"361 B"},{"name":"voice.json","shortname":"voice.json","datetime":"","size":"132 B"},{"name":"wifi.json","shortname":"wifi.json","datetime":"","size":"43 B"},{"name":"index.html.gz","shortname":"index.html.gz","datetime":"","size":"76.05 KB"}],"path":"/config","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""}
I notice the entry ,"mode":"direct", what is for ? also "status":"" is empty - is that normal ? should be ok
if no issue, no ?
EDIT2: looking at code I found you already fixed the issue in latest git I can see in scard.c , but as I cannot compile I cannot experiment if any other issue
if(pathlen > 1 && path[pathlen - 1] != '/') {
path[pathlen] = '/';
path[pathlen + 1] = '\0';
}
// char *fullname = ((file_server_data_t *)req->user_ctx)->scratch;
strcat(strcpy(fullname, path), filename);
I notice the entry ,"mode":"direct", what is for ?
I do not know, I guess it is from the WebUI 2 reverse engineering I did way back then. To be removed?
also "status":"" is empty - is that normal ? should be ok if no issue, no ?
That is a bug, fixed in the latest commit.
I have added the auto real-time report pushing but not yet with a run-time setting. Enable here. Interval is ms.
I have also added check for "webui" subprotocol in the websocket code, IMO this is a more robust way to telling grblHAL that the WebUI is charge than later checking for ESP commands arriving via HTTP. Is this something that you may consider to add when connecting to the websocket? It should work with other implementations as well as the supported subprotocols has to be reported back on connection accepted. The subprotocol could also be used for later versioning to ensure backwards compatibility.
To be removed?
is it up to you , webUI does not use it, I though it was used for another webUI
I have also [added check for "webui" subprotocol...you may consider to add when connecting to the websocket?
I need to see the impact - any change in general protocol / API will impact
So I need to check carefuly
is it up to you , webUI does not use it, I though it was used for another webUI
ok, I'll leave it for now.
I need to see the impact - any change in general protocol / API will impact
Should not be any in the short term (if the backend websocket implementation is compliant with RFC6455). It is a matter of just adding the string when connecting. Later it could be used for version checking, if the backend does not support the requested subprotocol then the WebUI could gracefully stop with an error message.
Ok I have tested latest git and now compilling !!!! Thank you I compiled with :
OPTION(WiFi "Wifi + protocols" ON)
OPTION(SDcard "SD Card Streaming" ON)
OPTION(WebUI "WebUI services" ON)
OPTION(WebAuth "WebUI authentication" OFF)
OPTION(SoftAP "Enable soft AP mode" ON) # WiFi only
I can see IP
[MSG:WIFI STA ACTIVE, IP=192.168.2.108]
But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list
now can list sub directory :+1:
but subdir creation failed due to missing /
if I am in www
directory and create a subdirectory subdir
I got wwwsubdir
created at root instead of www/subdir
Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date
ESP3D [ESP400] format.txt
if you change the network/sta
it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released
Sorry for path issue - it looks like rat hole issue 😓
I will work on adding the Host path:www
support to [ESP800] to check to see if it cover all needs, it should be ready when you will be back
But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list
Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?
Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date
Will fix. I have to improve password handling elsewhere too - I want to report passwords in clear when a local client is connected and not if a remote is.
if you change the network/sta it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released
I'll look into this later - do not change the trigger for now?
Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?
I use fresh download git according https://github.com/grblHAL/ESP32#how-to-build-using-esp-idf-v43 So using webUI provided no change but config And got issue...
Now I did:
and seems indeed make it work ...orz ... weird - I will keep watching, thank you for the feedback
I'll look into this later - do not change the trigger for now?
No need - just to confirm, I wait for your [ESP400] with all settings to work on grbHal panel, there is not need to spend time on $EG $ES $$
if you change to [ESP400]
because parser is different, and need to define new type syntax
I will test and adjust webUI for grblHal - enjoy you vaccations ^_^
EDIT: I had a look to your protocol suggestion and indeed it is good idea - I will work on this too
I have just committted a fix for wrong mapping of password fields and added masking of passwords. I have also fixed the CPU frequency reported by ESP420.
As for adding support for grblHAL datatypes float (decimal) is the most important - best if you can handle formatting/input masking, I know it is not easy if not supported by the framework. I'll try emulation for the others when I am back - I will be offline for a while...
best if you can handle formatting/input masking
I can support any type so it is not an issue I will update the docs for float / masks and exclusive masks which should be same as previous with on additionnal flag for first bit as exclusive
enjoy your offline time ^_^
I have updated / wrote the :
For new types : ESP400 format.md
It should cover all of your existing extra types
For websocket websocket.md
For connections values ESP800 Format.md
Let me know what you think 😸
@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.
Thanks, was actually the webui plugin that complained, may just be something minor...
Quick update, was a bumpy journey, but I seem to have WebUI 3 running well on my H7 port now..
The compilation issues turned out to be minor - some different variable names etc in the older FatFS version. But I've upgraded to the latest version anyway. That turned out to be the easy bit..
Was then having reliability problems with sdcard access on the Nucleo. This uncovered a new DMA/cache issue that I'd missed previously, and a separate signal integrity issue (using a breakout with the Nucleo) - which was resolved with a slower SDMMC peripheral clock.
But my biggest issue was the httpd service (mostly) failing to send data - which took me some time to debug. Long story short - I needed more Ethernet TX DMA descriptors. The way the http server builds a response with the dynamic headers, it uses up a descriptor for each header + a descriptor for the data.
Then just a final hiccup with needing the -u _printf_float
to get the sdcard files listed in WebUI, and is now happily running through imaginary jobs on the bench.. Nice!
@luc-github I have been busy working on adding a linux/unix style virtual filing system (VFS) and switching to that in the networking/webui/sdcard code. This means that "drives" are mounted as directories in the root file system - I'll have to work out how that maps to the WebUI. FYI I just commtited the first version of these changes.
I'll look into the websocket/ESP800/ESP400 changes, perhaps after I get Pico W networking up and running as this will allow me to mix filing systems in a developer friendly way (I am no fan of working with MCUs, such as the ESP32, lacking real debugging support). I'll come back to you when I have questions.
Pi Pico with networking and WebUI is up and running so time to move on...
Websocket subprotocol IMO this should be hardcoded at the client side, perhaps per target, and not returned in the ESP800 response from the backend. A minor issue is that you have webui-v3 in the memo and webui-V3 in the default ESP800 responses, these should match?
Datetimes The datetime format in the ESP800 and ESP140 requests should be in accordance with ISO8601?
Versioning This is tricky, since grblHAL (and other webui backend implementations) may lag your development pace and users might try to load an incompatible version it would be nice if this could be handled in a good way. Currently grblHAL can detect the client version from the websocket connection or from the version parameter supplied with the ESP800 request. I use this to select either the v2 or v3 command parser and I intend to add code to inform the client if the requested version is not supported. This means that either a ESP800 command or a websocket connection (with the appropriate subprotocol) is the first thing a client should attempt. Currently this is not the case for the WebUI client:
Would this be hard to change?
File system(s) handling I want to simplify this in the backend, will take some time...
Settings, float etc. I have not started looking at this yet.
Real time command response
grblHAL returns the streaming status in the real time command response, will it be possible to use this rather than polling the status via ESP701?
<Run|MPos:27.907,0.648,0.000|Bf:0,1023|FS:500,150|Pn:PXYZ|SD:0.1,pcb_zebra.gcode>
the [ESP800] is not handled per target - that is why I make it flexible - but if no info is available in ESP800 we can hard code a value per target - I am just not sure to see the point of it
about webui-V3 where do you see it ? in ESP3D I already implemented it and is lower case :
[ESP800]json
{
"cmd": "800",
"status": "ok",
"data": {
"FWVersion": "2.1.1-3.0.0.a212",
"FWTarget": "marlin",
"FWTargetID": "30",
"Setup": "Enabled",
"SDConnection": "shared",
"SerialProtocol": "Socket",
"Authentication": "Disabled",
"WebCommunication": "Synchronous",
"WebSocketIP": "192.168.2.178",
"WebSocketSubProtocol": "webui-v3",
"WebSocketPort": "81",
"Hostname": "Marlinesp3d",
"WiFiMode": "STA",
"WebUpdate": "Enabled",
"FlashFileSystem": "LittleFS",
"HostPath": "/",
"Time": "none"
}
}
Datetimes About format I used the simplest one YYYY-MM-DD-HH-mm-ss as for current usage, I did not saw the need of use a formated one, especially time is not keept currently when CNC/printer is off. And in WebUI V3 I did not implemented the display of file time yet to see if it was actually missing to someone - and got 0 feedback, so may be this option is useless and should be removed ?
About versioning it is your choice what you want to use, but to be honest I do not understand why need to be compatible with 2.1 especially that there are a lot of differences in commands, output format and web handle between both version, it is a 3.0 because it break compatibility with 2.1, it will be double work to maintain both IMHO, but this is your choice.
Real time status Yes of course no problem, this is how grbl_esp32 do, as mentioned before I did not understood why you used [ESP701] which is supposed for Flash streaming not SD in ESP3D but in case of Grbl_HAL which has the SD and is the consummer all stream should go SD stream using CNC Gcode command for hold / resume / stop IMHO implementing parsing of SD:0.1, pcb_zebra.gcode is not hard, 0.1 is progress on base 100 (0.1%) or base 1 (10%) ?
Also I have wrote more docs FYR : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/Memo/Handlers.md https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/Memo/Commands.md to clarify the API used by WebUI 3.0
the [ESP800] is not handled per target - that is why I make it flexible - but if no info is available in ESP800 we can hard code a value per target - I am just not sure to see the point of it
The point is that the client announces what protocol version it wants to talk. If set via ESP800 then the backend tells itself via the client - this kind of defeats the purpose of having it.
about webui-V3 where do you see it ?
Here is one example.
About format I used the simplest one YYYY-MM-DD-HH-mm-ss
You could have chosen ISO88601 formatting which is slightly different, e.g. 2022-09-06T20:17:16
. Not a big deal, I convert these to ISO8601 format by replacing the separators with the correct ones.
About versioning it is your choice what you want to use
I try to keep backwards compatibility when possible, there is at least one more webclient that I know of and not supporting 2.1 may break that.
implementing parsing of SD:0.1, pcb_zebra.gcode is not hard, 0.1 is progress on base 100 (0.1%) or base 1 (10%) ?
Great, the base is 100 (0.1%). grblHAL can also be configured to output the real time report automatically, currently as a compile time option. This can be used to disable polling via http requests. Is this of interest?
Also I have wrote more docs
Nice,
The point is that the client announces what protocol version it wants to talk. If set via ESP800 then the backend tells itself via the client - this kind of defeats the purpose of having it.
Yeah seems I took problem upside down - no issue I plan to do several update before the week end - I can remove the subprotocol from [ESP800] and hardcode it in webUI - it will also simplify code on FW side
Here is one example.
Ho that is test server and seems ok - but I will update it as mentioned above - so it will be removed
2022-09-06T20:17:16
To be honest I just put -
because I though will be easier to parse...but looking at code now I do not see the benefit...
as I will update I can follow the format you mention - to be honest did not do any research on ISO format when I implemented it
I try to keep backwards compatibility when possible
Ok
This can be used to disable polling via http requests. Is this of interest?
Yes web UI parse all answers even no polling command to handle auto reporting, auto reporting is better if well managed on FW side , like if it does not pop in another running answer
to be honest did not do any research on ISO format when I implemented it
Well, you were pretty close so you must have seen it before. I have created a c function for parsing the string that you may use if you want.
Yes web UI parse all answers even no polling command to handle auto reporting, auto reporting is better if well managed on FW side , like if it does not pop in another running answer
Yes, I saw that, even the $G
response is handled when sent automatically (grblHAL can be configured to send it on changes).
From your commands memo it looks like json output is now the default, is this correct?
Also many commands has pwd
as a parameter. How/when is this intended to be used? When not logged in (no cookie present) and authentication is enabled?
I have created a c function for parsing the string
Thank you I will have a look
it looks like json output is now the default
The webUI only use json yes, in ESP3D default output is plain text because if you type command in Serial/Telnet it for human reading and add json=yes
to the command change output to json which is formated response for webUI, only ESP400 does not have plain text because I was lazy to do it as the plain text would not be really human readable anyway
many commands has pwd as a parameter.
Yes: the pwd=admin/user
parameter is only for serial/telnet when authentication is enabled, http use the cookie once authenticated instead
That said for serial ESP3D has 2 anothers settings :
I've just committed an update for grblHAL with changes mostly for WebUI v3 support.
ESP400 now outputs all grblHAL settings, I emulate bitfields and axismasks via B
type settings - this may change later when native support is available in the WebUI client.
Flash filing system implemented by using littlefs for ESP32, RP2040 and iMXRT1020 drivers. I chose littlefs as a standard flash filing system for grblHAL since the lowlevel driver code is fairly simple if the target MCU has an API for handling flash write/read/erase, and also since it was faily easy to add last modified datestamp to files in the VFS wrapper. @dresco you may consider adding this to the STM32H7xx driver - there is ample flash available for that(?) and only a smallish low-level driver is needed, here is the one used for RP2040.
Improved search across mounted file systems (or drives) for index.html.gz, favicon.ico, preferences.json and, for WiFi builds, _aplogin.html. Added support for http://<ip>/?forcefallback=yes
, this will attempt to load index.html.gz from the VFS /embedded folder/mount. Note that this currently fails due to the client not understanding the ESP800 data provided by the backend. I see that the ESP800 request does not add json=yes
parameter, should the response be in plain text or json? If I return json I still get an error...
A question: is it per design that it is not possible to stream gcode files from flashfs? (flashfs is mapped to a littlefs mount directory in grblHAL).
ESP400 now outputs all grblHAL settings, I emulate bitfields and axismasks via B type settings - this may change later when native support is available in the WebUI client
bit maks support is ready in code but not implemented in ESP400 I will add it soon
Flash filing system implemented by using littlefs
This the prefered choice also in ESP3D/ESP3DLib
I see that the ESP800 request does not add json=yes
all ESP command used by web UI need json=yes commands
You may use an old version of webUI if json=yes is no used
A question: is it per design that it is not possible to stream gcode files from flashfs? (flashfs is mapped to a littlefs mount directory in grblHAL).
Yes it is possible, I do not encourage this because of flash wearing and small size of flash, I can add it if you really want but I am confused I though you hide the flash partition GrblHal can have custom Flash it is possible, currently all targets use same but I can add custom one for GRBHal like for SD https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/FLASH-source.js#L24 https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/DIRECTSD-source.js The webUI is flexible ^_^
I will give a try to your latest version this week and check
@terjeio I have do a fresh download of everything (https://github.com/grblHAL/ESP32#how-to-build-using-esp-idf-v43) and cannot compile :
no-stringop-truncation -MD -MT esp-idf/main/CMakeFiles/__idf_main.dir/webui/server.c.obj -MF esp-idf\main\CMakeFiles\__idf_main.dir\webui\server.c.obj.d -o esp-idf/main/CMakeFiles/__idf_main.dir/webui/server.c.obj -c ../main/webui/server.c
../main/webui/server.c: In function 'wifi_scan_handler':
../main/webui/server.c:424:32: error: 'file' undeclared (first use in this function); did you mean 'pipe'?
vfs_puts(resp, file);
^~~~
pipe
../main/webui/server.c:424:32: note: each undeclared identifier is reported only once for each function it appears in
../main/webui/server.c: In function 'wifi_login_receive_finished':
../main/webui/server.c:489:31: error: 'file' undeclared (first use in this function); did you mean 'pipe'?
vfs_puts("Connecting...", file);
^~~~
pipe
[1329/1334] Building C object esp-idf/main/CMakeFiles/__idf_main.dir/webui/commands_v3.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1
C:\Users\luc\Documents\GRBLHal\esp\esp-idf\ESP32>
Sorry I am out today and did not dig in issue much
As started to discuss here https://github.com/grblHAL/ESP32/issues/39