search

luc-github / ESP3D-WEBUI

A Web UI for ESP8266 or ESP32 based boards connected to 3D printers / CNC
GNU General Public License v3.0
747 stars 305 forks source link

[FEATURE REQUEST]Add grblHAL support : https://github.com/grblHAL #252

Closed luc-github closed 1 year ago

luc-github commented 2 years ago

As started to discuss here https://github.com/grblHAL/ESP32/issues/39

terjeio commented 2 years ago

An initial update for WebUI 3.0 of the grblHAL WebUI plugin is now available. I am testing with a Nucleo STM32F756 board and the plugin should also work with the iMXRT1062 (Teensy 4.x) driver. Note that the ESP32 grblHAL driver does not use the plugin, I will backport the code changes to that later on.

@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.

luc-github commented 2 years ago

Ok understood - unfortunatly I have no STM boards, neither Teensy 4.x, but I can emulate webserver and board responses in webUI dev mode

I saw the the https://github.com/grblHAL/Plugin_WebUI but I was thinking was same code in https://github.com/grblHAL/ESP32 so I did not realized it was V3 based - I will have a look

luc-github commented 2 years ago

@terjeio you wrote

my plan to contact you about some issues I have encountered

Can you tell which ones ? I can start by these first

I have added the code base to test and build grblHAL webUI https://github.com/luc-github/ESP3D-WEBUI/commit/bac40f99ee64e1ed310ba924acc120e5f1bcf166 image

I need some confirmation :

I will check what play SD job commands you are using, reading https://github.com/grblHAL/core/wiki/SD-Card-implementation,-a-technical-walktrough it seems $F=<filename but I will double check

terjeio commented 2 years ago

Issues encountered:

ESP701 for querying the status of a running job suddenly (randomly) is no longer sent after a running a few. Reloading resumes the querying.

If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing.

The default file system is FLASHFS, could this be the first listed in the target supportedFileSystems (in files.js) or perhaps read from the ESP800 response?

https://github.com/luc-github/ESP3D-WEBUI/blob/bac40f99ee64e1ed310ba924acc120e5f1bcf166/src/components/Panels/Files.js#L509-L512

I've changed the code above and it seems to work but not sure if this has any side-effects (other than failure if none are specified):

        if (useUiContextFn.getValue("autoload") && currentFS == "") {
            currentFS = files.supported[0].value
            onSelectFS()
        }

Is grblHal typo correct ? I saw also different writing like GRBLHal and GRBLHAL

"grblHAL" is the correct version. The welcome message from the controller is not correct, but I have been reluctant to change it as it may break senders.

do you have a logo for grblHal

There are a few proposals but none has been decided upon. I have to make my mind up...

Do you want to go to https://github.com/grblHAL/ ?

Yes, I have changed this already.

I see on esp32 version you cannot update the webUI as do not use the maintenance page and directly flashed the webUI, is that the same with pluggin ?

Yes. For the plugin changing this will require a flash filing system. I may add this later. If the WebUI is compiled into the binary then adding index.html.gz to the SD card www folder will override the one from the binary so there is an update path available. More work on this is needed on my end - perhaps adding download of a new version to the www folder as a first step?

I will check what play SD job commands you are using

I am using $F=, I can also use ESP220 (IIRC this is used by WebUI 2.1) but when I check your ESP3D source it seems to me that this (and some other commands) has been repurposed? The commands that may have been reporposed have the help text "??" in this list.

I see extra real time commands: ... Is that list complete / up to date ?

I am not sure it is up to date, I'll have to check. Most can be ignored? Perhaps the most important change is the new Tool state, it would be great if WebUI could support grblHAL style tool change handling. And yes, this works even when running a job from the SD card.

Here are the preferences and target folder I created:

preferences.zip

grblHAL.zip

Should grblHAL get its own firmware target name and id?

Some ideas:

Can WebUI fetch the settings labels from the controller? grblHAL has dynamic settings handling - plugins may add new ones, even user defined plugins. In my sender the settings UI is built from information requested from the controller. I can easily add an ESP command for fetching a json formatted list of setting ids and labels (and even descriptions) in a json formatted response.

On a longer term todo list will it be possible to use the WebUI style settings handling? It will require the support for floats and a flag type where bit 0 controls the enable status for the rest.

luc-github commented 2 years ago

ESP701 for querying the status of a running job

Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel

If wrong password is given on login correcting it and resubmitting leads to the USER field in the form data missing. I need to check but I may did that on purpose to display or not the error message

The default file system is FLASHFS,

This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet

I am using $F=, I can also use ESP220

Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0

I am not sure it is up to date, I'll have to check. Most can be ignored

it is just to allow these command to be used by macro using a text descriptor as they are not printable,

I also see some commands : ?,~,!` need to be update by the not printable code to be handled properly in grblHal, so will do

would be great if WebUI could support grblHAL style tool change handling.

You will have to tell me more what you expect for UI and behavior, but I see no issue to support it

target name and id?

The current webUI use grblhal as name, for the ID I did not defined it yet, but because it is not selectable for ESP3D, it is not major issue, here the current list : https://github.com/luc-github/ESP3D/blob/3.0/esp3d/src/include/defines.h#L28-L36 If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW

Can WebUI fetch the settings labels from the controller? currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language

So actually you can add any sections / items you want in settings just follow current syntax, if need more type of settings, just need to define type, type constraint, rendering form and should be ok like others types

luc-github commented 2 years ago

Reading your version now

luc-github commented 2 years ago

I have updated and build the index.html.gz according your changes https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/dist/CNC/GRBLHal/index.html.gz

I still need to update the real commands parts

I have also noticed you do not support the one session feature so many webUI can be open at once, is that a choice ? I won't recommend it

terjeio commented 2 years ago

ESP701 for querying the status of a running job

Currently this command is for the ESP3D Gcode host not used for running job, why not use the ? like GRBL_ESP32 ? I need to see your implementation, originally ESP command are send by http and answer come back by http too, unlike others commands which use websocket, if polling is stopped it means there is a crash that should be visible in dev console or an error in notifications panel

Polling for the real time status continues, only the ESP701 request stops. If I remove the ESP701 request the stream status is not displayed even if it is available in the real-time report. I assume this could be changed to use the real time report instead?

The default file system is FLASHFS,

This is already the case, no ? Flash is the default because it is the one I know it is present for sure, so not sure what issue you meet

It is not present in the controllers using the plugin.

I am using $F=, I can also use ESP220

Yes these commands are used by GRBL_ESP32 which is only supported by WebUI 2.1, and since the GRBL_ESP32 is dropped, I did not maintained them in WebUI 3.0

Ok, this explains it. They should not have been reassigned to a different use then? Or was this a customization made by bdring and his team?

target name and id?

If you are ok to use ID=80 I will set it, it will be used to be sure WebUI match the target FW

If not needed I am ok with using ID=10 as I do not know what it is used for in your code.

Can WebUI fetch the settings labels from the controller?

currently it is the case, everything is dynamic, these labels are then translated by translation tool to give better look and correct language

Ok, what is the ESP command to use for returning those. And to be clear I am talking about the grblHAL settings where those not known by the WebUI does not have useful labels:

image

preferences content has some changes already, polling commands can have difference frequency

Ok, will update. Sun came out today after a period of rain - so it may take some time. And I will be away for a period starting Tuesday 2nd.

I have also noticed you do not support the one session feature so many webUI can be open at once, is that a choice ?

Definitely not - or at least until an abitration protocol can be added. Only one client should be in control at the time. As a first step I might add support for additional read-only clients - but this is longer term.

luc-github commented 2 years ago

It is not present in the controllers using the plugin.

I need to check if I have added a dependency on ESP800 settings 

 [ESP800]
{
 "cmd": "800",
 "status": "ok",
 "data": {
  "FWVersion": "3.0.0.a111",
  "FWTarget": "grblhal",
  "FWTargetID": "80",
  "Setup": "Enabled",
  "SDConnection": "direct",
  "SerialProtocol": "raw",
  "Authentication": "Disabled",
  "WebCommunication": "Synchronous",
  "WebSocketIP": "localhost",
  "WebSocketPort": "81",
  "Hostname": "esp3d",
  "WiFiMode": "STA",
  "WebUpdate": "Enabled",
  "FileSystem": "LittleFS",
  "Time": "none",
 }
}

grblHAL settings

these are in language pack - they are just translation from commands id : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/translations/en.json

the one session feature

this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want

I need to setup an esp32 system for grblHal as I am afraid I will do some blind code and may misunderstood some commands

if I replace the plugin files to existing ESP32 code would it work ? any other changes should I do - I can do changes manually and this will let me know also why current version is always in reset error on my board

Or was this a customization made by bdring and his team

In FluidNC they renamed all commands https://github.com/bdring/FluidNC/wiki/FluidNC-Commands-and-Settings

dresco commented 2 years ago

@dresco is working on a STM32H7xx driver that has ethernet and SD card support, it will be possible to use the plugin for this driver when it is updated to use the latest grblHAL core.

Just fyi, have started looking at this. First issue is that I used the STMCubeIDE provided middleware for SD card access (using DMA & SDMMC peripheral) - which uses an older FatFs version. Am looking into what it's going to take to upgrade..

@terjeio should I aim for R0.13c as per F4/F7 etc, or latest R0.14b?

terjeio commented 2 years ago

the one session feature

this feature is actually mostly handled by webUI, the FW just broadcast the last connected ID by websocket , so the old ID webUI automaticaly close - not hard to implement - I may try to do it if you want

Wait... One step at a time please. I have some support issues to clear before I can dig into the details.

if I replace the plugin files to existing ESP32 code would it work ?

No, it is a major refactoring. Ideally I should update the ESP32 driver and the plugin so it could be used directly - I do not like to maintain duplicate code.

why current version is always in reset error on my board

Oops, I forgot to asnwer this, by default grblHAL uses normally closed switches for all inputs. Connect the reset pin to ground and you should be fine.

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

dresco commented 2 years ago

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

Thanks, was actually the webui plugin that complained, may just be something minor...

luc-github commented 2 years ago

WebUI is updated to a7

Now will check : https://github.com/grblHAL/core/wiki/Report-extensions because additional grblHal states are not supported by webUI currently

luc-github commented 2 years ago

about : Can WebUI fetch the settings labels from the controller? it is possible just need to define the format you want to use and I can customize the grblHal panel for settings because it is specific, currently I use the GRBL $$ display but we can imagine another command that do same and add label and constraint for each like for ESP3D settings even section if necessary and it will dynamically build the UI accordingly

terjeio commented 2 years ago

I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin. Add preferences.json to the root of the SD card.

I left the SerialProtocol value to Socket in the ESP800 response - not sure ift that is correct but it works for me. There is flash filesystem support for the ESP32, I will change the Filesystem value later to reflect that.

Note that forms based autentication is not yet supported for the ESP32 driver. I'll put improved settings handling down the list until the basic stuff is completed - forms based login and unified SD card handling are the largest items.

And BTW the ESP32 webserver is really slow compared to e.g. the iMXRT1062 controller where WebUI loads in the blink of an eye...

luc-github commented 2 years ago

I have just committed an update to the ESP32 driver that uses (parts of) the WebUI plugin.

Ok thank you I give a try tomorrow - I was out all day today and need to finish some code first

SerialProtocol value to Socket in the [ESP800 response] - it is fine - there is no use currently in WebUI but better to be consistent, if the need popup

Ho yes ESP32 sync webserver is very slow, the test I did on current grblHal is already far faster (it use the idf async http server right ?) I am not surprised on another chip the performances are better - on reprap they claim to have wifi file transfer of 1.2MB/s but they do not use esp webserver ... I have also noticed performances issues related to SD https://github.com/luc-github/ESP3D/issues/576, so I use SDfat instead of ESP32 SD library in ESP3D

thank you for the porting It will be easier for me to experiment - I will feedback you tomorrow

luc-github commented 2 years ago

Hi I have flashed my board with latest git and webui 3.0, very cool job thank you,

so I have some questions :

3.0.txt 2.X.txt

Sorry to ask so many questions, hope you can answer before you will going to vacations - so I won't bother you 😸

Edit I see: $ES so I may use it instead of $$ to list grblHal settings, is that ok ? change command still seems same $<ID>=<value>, right ? Edit2: if I read well need to send $EG then $ES then $$ to be able to display settings properly ? any reason not to add the value in $ES ? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of $ES line format, sender knowing $ES command should not need the $$, because $$ would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway

terjeio commented 2 years ago

Is that a choice to not display the IP at start ?

Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.

This is the latest v3 json version:

ESP800

{"cmd":"800","status":"ok","data":{"FWVersion":"1.1f","FWTarget":"grblHAL","FWTargetID":"80","Setup":"Enabled","SDConnection":"direct","SerialProtocol":"Socket","Authentication":"Enabled","WebCommunication":"Synchronous","WebSocketIP":"10.0.0.22","WebSocketPort":"81","Hostname":"Grbl","WiFiMode":"STA","WebUpdate":"Disabled","FileSystem":"none","Time":"none"}}

Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested): http://10.0.0.22/command?cmd=[ESP800]json=yes%20version=3.

the CPU freq is 19 - I guess this is wrong

Yep, it is the stepper code clock frequency, I have to add a new HAL value for the MCU clock...

$G ... what is the meaning of . in S0. ?

I guess it is for implying that the value is a float - inherited from legacy Grbl.

Form another side, is that normal to have F0 instead of F0.0

Good question, I have to check my code.

I see $ES so I may use it instead of $$ to list grblHal settings, is that ok ?

No, $ES is for getting information about the settings, not the values. There is an internal API for querying setting details which I use to generate the ESP400 response, for most grblHAL settings this is relatively straight forward. The settings API is designed to allow user defined settings (in plugins or board support code), these are also possible to query and will show up in the $ES response etc.

change command still seems same $<ID>=<value>, right ?

Correct. I try to be backwards compatible with the legacy Grbl specification, at least in spirit.

if I read well need to send $EG then $ES then $$ to be able to display settings properly ?

I would rather like to add a ESP command for querying setting details, same format as ESP400. I can even add most to ESP400 by just looping trough them instead of outputting those that your backend outputs.

any reason not to add the value in $ES ? it make a lot of queries and parsing that could at least be simplified by just adding value at last position of $ES line format, sender knowing $ES command should not need the $$, because $$ would be only for old senders for compatibility IMHO, I will handle the multiple queries anyway

The reason is that $ES is used once at startup (at least in my sender) to build the settings UI. $$ is then used to fetch the values as needed.

luc-github commented 2 years ago

Note that the backend supports v2.1 as well as v3.0, I use the version parameter added to the ESP800 command to detect 3.0 and set the mode for all subsequent commands (until a new ESP800 is requested):

Ok so I see the issue - I have manually sent [ESP800] in terminal to verify settings when checking the File system:directsd issue and that generate the problem

I would rather like to add a ESP command for querying setting details, same format as ESP400

The ESP400 management is not part of specific FW but common to all target ESP3D ESP3DLib GRBL_ESP32 grblHal etc...

if you want to merge all settings in ESP400 like format - I can remove the tab Features for grblHal, and handle these settings in grblHal tab, which give you all flexibility for the additional types, I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX for extended settings display @terjeio What do you think ?

Yes, as it cannot be displayed for all the platforms grblHAL supports, see this wiki page. I may add SSDP and/or add the IP address to the full status real-time report to fix this.

My question was because you already display [MSG:WIFI STA ACTIVE] so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO

When locked the ? seems not working so I am not sure putting IP in report is necessary, additionnaly it will add extra info that should not change, and this info is already in $I

Final question : what sender currently support all/most grblHal extra features ? I have tried https://github.com/terjeio/ioSender/releases/download/2.0.40/ioSender.2.0.40.zip but it does not work on my Win11 and My norton AV does not like it, more over it remove each part of the sofware one by one saying there is a problem

This is mostly to have a reference to verify I interpret properly the new states in webUI

terjeio commented 2 years ago

I would prefer not using [ESP400] command to avoid confusion if possible but something like $EX for extended settings display @terjeio What do you think ?

IMO it will be better to add a new ESP command that outputs the information in json format, possibly the same as ESP400 uses as you already have parser/UI builder code for that.

My question was because you already display [MSG:WIFI STA ACTIVE] so adding [MSG: IP XXX.XXX.XXX.XXX] when is possible is still better than nothing by hooking the get Ip event I think IMHO

I can add that but there will still be a problem with some controllers that uses native USB for communication as most of these loses any info sent before a client is connected. Not easy this...

When locked the ? seems not working

It should work - are you sure about this.

Final question : what sender currently support all/most grblHal extra features ?

Only ioSender as far as I know.

... but it does not work on my Win11 and My norton AV does not like it

Is it Norton blocking it? I have not signed the code as the certificate for doing so is expensive.

This is mostly to have a reference to verify I interpret properly the new states in webUI

There is only one new state Tool in the real-time report, Alarm and Run can have substates added if enabled in settings. These substates are added in the same way as for Hold. The $G report has some missing parser states added as well as some new ones.

luc-github commented 2 years ago

IMO it will be better to add a new ESP command

if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings

It should work - are you sure about this.

I have reflashed to test the fixes in webUI I did and I cannot duplicate anymore ... I am stumped...

Is it Norton blocking it?

yes and deleting all files one by one, I have paused Norton and now everything seems working - only the SD tab seems unresponsive but it is working in webuI so - it is not a problem for me

The $G report has some missing parser states added as well as some new ones

Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 ___M5 M9 T0 F0 S0 _] forGRBL

Forget my comment about F0.0, it seems my GRBL fw do alsoF0instead ofF0.0`

I will check everything - thank you for all feed back and enjoy your vacations, I have everything to work now I think - this will keep me busy for a couple of days ^_^

terjeio commented 2 years ago

IMO it will be better to add a new ESP command

if you really want to use ESP command then keep ESP400, the parser need to be modified for the new types only - the command itself is not an issue if you use json format - the parsing is piece of cake, in all case I will remove the feature tab as it is duplicate with grblHal settings

Yes - I guess this makes it simpler for both me and you. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask) and 6 (decimal or float if you will, later I might add double precision support in the core as an option...).

Examples in ioSender:

$22 - homing enable (exclusive bitfield) - can be emulated imperfecly by your B type:

image Disabled

image Enabled

$5 - invert limit pins (axismask)

image IMO this can be emulated by your B type so not really unsupported.

Yes I need to add the news states [GC:G0 G54 G17 G21 G90 G94 G49 G98 G50 M5 M9 T0 F0 S0. ] for GrblHal [GC:G0 G54 G17 G21 G90 G94 ___M5 M9 T0 F0 S0 _] forGRBL

There are many more new ones, here is the source code in report.c and motion modes supported in gcode.h. grblHAL also supports the G59.1, G59.2 and G59.3 offsets.

If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller? Note that these can also be added to by third party code.

luc-github commented 2 years ago

. The datatypes not directly supported by WebUI are 2 (exclusive bitfield), 4 (axismask)

ho yes mask controls are a pain - I had some before and I converted all them to simple booleans, but I will see what I can do

If you aim for perfection mapping of error and alarm codes to text should be pulled from the controller?

No I am not - I will keep this part for then end because anyway all text need to be in translations packs so I currently just refer to code for that

There are many more new ones, here is the source code

Yes saw it - thank you

terjeio commented 2 years ago

I have committed some updates, noatbly forms based login (as used by WebUI 3.0) should now work and I have added IP address to the WIFI STA ACTIVE message.

luc-github commented 2 years ago

Ok will check that - I am still working on sanity check if not Flash is enabled - it was not a case I had planned so I am following every rat holes to be sure I did not forget anything

that bring me back to my question :

[ESP800] File system:"directsd" directsd is not supported should be none or spiffs or littlefs

Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www, So add a flag hostFS: "SD/www" in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?

Note: I have prepared a code to handle Bit Masks, but I will wait your decision of the format of [ESP400] with all settings to implement corresponding controls

terjeio commented 2 years ago

[ESP800] File system:"directsd" directsd is not supported should be none or spiffs or littlefs

none is a bit counterintuitive, could virtual be a more logical option?

Above status lead me to add a new flag : hostFS, because you mentioned you do not use Flash on some system, so the index.html.gz is hosted on SD (or USB may be in future), so WebUI update, preferences.json should be read/uploaded from/to SD/www,

In fact it is possible to fetch the WebUI from flash, but then from a read only file system. I do check if the file is available in the www folder and if so it will used instead.

So add a flag hostFS: "SD/www" in ESP800 answer would tell webUI where to load/save preferences.json, what do you think ?

That could work, even when file system virtualization is implemented. Perhaps hostFiles or hostPath is a better name? Which file system becomes irrelevant when virtualization is implemented?

Just for the fun of I added a few lines to auto push the real time status report (along the $G-parser state report if enabled and changes to the report) and it works!

Just add this code before webui_init() if you want to test it:

static on_execute_realtime_ptr on_execute_realtime;

static void auto_report (sys_state_t state)
{
    static uint32_t ms = 0;

    if(hal.get_elapsed_ticks() >= ms) {
        ms = hal.get_elapsed_ticks() + 3000; //ms
        if(hal.stream.state.webui_connected)
            protocol_enqueue_realtime_command(CMD_STATUS_REPORT);
    }

    on_execute_realtime(state);
}

and in webui_init() add:

    on_execute_realtime = grbl.on_execute_realtime;
    grbl.on_execute_realtime = auto_report;

Change $10 to enable auto-reporting of the parser state:

image

If this is of interest there should be a way to set the timeout?

luc-github commented 2 years ago

File system:"directsd"

Well the wording is confusing now I should admit, initially it was meant for Flash only, as I did not expected Falsh was not available - I may rename it to FlashFS instead for clarity

In fact it is possible to fetch the WebUI from flash, but then from a read only file system.

For this part it does not really matter where the original webUI is located as server already streamed it and it is loaded on client side, unless you want a read only preferences which make no sense I think - if FS is not accessible it is better to say there is no Flash FS available IMHO, it avoid user confusion and limit user interaction (and unexpected situation)

That could work, even when file system virtualization is implemented. Perhaps hostFiles or hostPath is a better name?

I think hostPath is fine if on SD it would be SD/www as done today, indeed only location is important, not FS actually

If this is of interest there should be a way to set the timeout? well if you put a time out configurable then you can use the timeout to disable the autoreporting using value of timeout to 0 and no need to put a flag in report option, just my 2 cents, another solution is no timeout but send report each time a command is received executed, or status change, but that may be more complex to implement

I will give a try tomorrow to this new version thank you 😸

Edit: I see your / to serve files is on / of SD so hostPath: www should be ok, I must also change the path for SD files to allow download by removing SD/

luc-github commented 2 years ago

Unfortunatly I cannot compile latest git I have raise issue https://github.com/grblHAL/ESP32/issues/40 I found some sanity check are missing but still have issues

I have also did some test on current version to test the upload of preferences.json and found that any SD sub folder cannot be parsed or even have some weird behavior listing the root content sometimes,

Upload to these sub directories seems working as I got [MSG:Upload ok] and I can download the file if I write proper path in browser, but listing is not working and this is not limited to www subdirectory but all subdirectory is that normal ?

EDIT: I found the root cause of parsing issue of subfolders command sent to grblHal : http://192.168.2.108/sdfiles?path=%2Fconfig&action=list&t=1659321435917

give : {"files":[],"path":"/confi","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""} which is wrong should be config not confi the last char seems trailed confirmed by double it : http://192.168.2.108/sdfiles?path=%2Fconfigg&action=list&t=1659321435917 which give better output : {"files":[{"name":"pen_position.json","shortname":"pen_position.json","datetime":"","size":"24 B"},{"name":"readme.txt","shortname":"readme.txt","datetime":"","size":"361 B"},{"name":"voice.json","shortname":"voice.json","datetime":"","size":"132 B"},{"name":"wifi.json","shortname":"wifi.json","datetime":"","size":"43 B"},{"name":"index.html.gz","shortname":"index.html.gz","datetime":"","size":"76.05 KB"}],"path":"/config","total":"3.71 GB","used":"44.59 MB","occupation":"1","mode":"direct","status":""}

I notice the entry ,"mode":"direct", what is for ? also "status":"" is empty - is that normal ? should be ok if no issue, no ?

EDIT2: looking at code I found you already fixed the issue in latest git I can see in scard.c , but as I cannot compile I cannot experiment if any other issue 

if(pathlen > 1 && path[pathlen - 1] != '/') {
            path[pathlen] = '/';
            path[pathlen + 1] = '\0';
        }

//        char *fullname = ((file_server_data_t *)req->user_ctx)->scratch;

        strcat(strcpy(fullname, path), filename);
terjeio commented 2 years ago

I notice the entry ,"mode":"direct", what is for ?

I do not know, I guess it is from the WebUI 2 reverse engineering I did way back then. To be removed?

also "status":"" is empty - is that normal ? should be ok if no issue, no ?

That is a bug, fixed in the latest commit.

I have added the auto real-time report pushing but not yet with a run-time setting. Enable here. Interval is ms.

I have also added check for "webui" subprotocol in the websocket code, IMO this is a more robust way to telling grblHAL that the WebUI is charge than later checking for ESP commands arriving via HTTP. Is this something that you may consider to add when connecting to the websocket? It should work with other implementations as well as the supported subprotocols has to be reported back on connection accepted. The subprotocol could also be used for later versioning to ensure backwards compatibility.

luc-github commented 2 years ago

To be removed?

is it up to you , webUI does not use it, I though it was used for another webUI

I have also [added check for "webui" subprotocol...you may consider to add when connecting to the websocket?

I need to see the impact - any change in general protocol / API will impact

So I need to check carefuly

terjeio commented 2 years ago

is it up to you , webUI does not use it, I though it was used for another webUI

ok, I'll leave it for now.

I need to see the impact - any change in general protocol / API will impact

Should not be any in the short term (if the backend websocket implementation is compliant with RFC6455). It is a matter of just adding the string when connecting. Later it could be used for version checking, if the backend does not support the requested subprotocol then the WebUI could gracefully stop with an error message.

luc-github commented 2 years ago

Ok I have tested latest git and now compilling !!!! Thank you I compiled with :

OPTION(WiFi "Wifi + protocols" ON)
OPTION(SDcard "SD Card Streaming" ON)
OPTION(WebUI "WebUI services" ON)
OPTION(WebAuth "WebUI authentication" OFF)
OPTION(SoftAP "Enable soft AP mode" ON) # WiFi only

I can see IP [MSG:WIFI STA ACTIVE, IP=192.168.2.108]

But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list now can list sub directory :+1: but subdir creation failed due to missing / if I am in www directory and create a subdirectory subdir I got wwwsubdir created at root instead of www/subdir

Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date ESP3D [ESP400] format.txt if you change the network/sta it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released

Sorry for path issue - it looks like rat hole issue 😓

I will work on adding the Host path:www support to [ESP800] to check to see if it cover all needs, it should be ready when you will be back

terjeio commented 2 years ago

But I also see a bug related to path, I saw it before but I though the fix mentioned above will fix it also so I did not want to overflow the issue list

Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?

Also I can see the [ESP400] output is incorrect for password - it is set to byte/boolean and it is sent in clear, I wrote a note that should be up to date

Will fix. I have to improve password handling elsewhere too - I want to report passwords in clear when a local client is connected and not if a remote is.

if you change the network/sta it won't display the scan network button, this is minor issue - as it will be custom ESP400 I can change the trigger - but once defined it is better not to change until major revision is released

I'll look into this later - do not change the trigger for now?

luc-github commented 2 years ago

Odd as this is working for me - I can create subdirectories no problem. Try clean the project and recompile?

I use fresh download git according https://github.com/grblHAL/ESP32#how-to-build-using-esp-idf-v43 So using webUI provided no change but config And got issue...

Now I did:

and seems indeed make it work ...orz ... weird - I will keep watching, thank you for the feedback

I'll look into this later - do not change the trigger for now?

No need - just to confirm, I wait for your [ESP400] with all settings to work on grbHal panel, there is not need to spend time on $EG $ES $$ if you change to [ESP400] because parser is different, and need to define new type syntax

I will test and adjust webUI for grblHal - enjoy you vaccations ^_^

EDIT: I had a look to your protocol suggestion and indeed it is good idea - I will work on this too

terjeio commented 2 years ago

I have just committted a fix for wrong mapping of password fields and added masking of passwords. I have also fixed the CPU frequency reported by ESP420.

As for adding support for grblHAL datatypes float (decimal) is the most important - best if you can handle formatting/input masking, I know it is not easy if not supported by the framework. I'll try emulation for the others when I am back - I will be offline for a while...

luc-github commented 2 years ago

best if you can handle formatting/input masking

I can support any type so it is not an issue I will update the docs for float / masks and exclusive masks which should be same as previous with on additionnal flag for first bit as exclusive

enjoy your offline time ^_^

luc-github commented 2 years ago

I have updated / wrote the : For new types : ESP400 format.md
It should cover all of your existing extra types

For websocket websocket.md

For connections values ESP800 Format.md

Let me know what you think 😸

dresco commented 2 years ago

@dresco - the SD card plugin can work with earlier versions of FatFS, I should have used the version id provided by FatFS but are using the target MCU family for now for detection... If you update aim for the latest version.

Thanks, was actually the webui plugin that complained, may just be something minor...

Quick update, was a bumpy journey, but I seem to have WebUI 3 running well on my H7 port now..

The compilation issues turned out to be minor - some different variable names etc in the older FatFS version. But I've upgraded to the latest version anyway. That turned out to be the easy bit..

Was then having reliability problems with sdcard access on the Nucleo. This uncovered a new DMA/cache issue that I'd missed previously, and a separate signal integrity issue (using a breakout with the Nucleo) - which was resolved with a slower SDMMC peripheral clock.

But my biggest issue was the httpd service (mostly) failing to send data - which took me some time to debug. Long story short - I needed more Ethernet TX DMA descriptors. The way the http server builds a response with the dynamic headers, it uses up a descriptor for each header + a descriptor for the data.

Then just a final hiccup with needing the -u _printf_float to get the sdcard files listed in WebUI, and is now happily running through imaginary jobs on the bench.. Nice!

luc-github commented 1 year ago

@terjeio I see you are back - I have implemented your suggestion for websocket subprotocol in webUI V3, and wrote all docs for websocket/ ESP800/ ESP400 let me know if you need me to do anything

terjeio commented 1 year ago

@luc-github I have been busy working on adding a linux/unix style virtual filing system (VFS) and switching to that in the networking/webui/sdcard code. This means that "drives" are mounted as directories in the root file system - I'll have to work out how that maps to the WebUI. FYI I just commtited the first version of these changes.

I'll look into the websocket/ESP800/ESP400 changes, perhaps after I get Pico W networking up and running as this will allow me to mix filing systems in a developer friendly way (I am no fan of working with MCUs, such as the ESP32, lacking real debugging support). I'll come back to you when I have questions.

terjeio commented 1 year ago

Pi Pico with networking and WebUI is up and running so time to move on...

image

Would this be hard to change?

luc-github commented 1 year ago
luc-github commented 1 year ago

Also I have wrote more docs FYR : https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/Memo/Handlers.md https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/Memo/Commands.md to clarify the API used by WebUI 3.0

terjeio commented 1 year ago

the [ESP800] is not handled per target - that is why I make it flexible - but if no info is available in ESP800 we can hard code a value per target - I am just not sure to see the point of it

The point is that the client announces what protocol version it wants to talk. If set via ESP800 then the backend tells itself via the client - this kind of defeats the purpose of having it.

about webui-V3 where do you see it ?

Here is one example.

About format I used the simplest one YYYY-MM-DD-HH-mm-ss

You could have chosen ISO88601 formatting which is slightly different, e.g. 2022-09-06T20:17:16. Not a big deal, I convert these to ISO8601 format by replacing the separators with the correct ones.

About versioning it is your choice what you want to use

I try to keep backwards compatibility when possible, there is at least one more webclient that I know of and not supporting 2.1 may break that.

implementing parsing of SD:0.1, pcb_zebra.gcode is not hard, 0.1 is progress on base 100 (0.1%) or base 1 (10%) ?

Great, the base is 100 (0.1%). grblHAL can also be configured to output the real time report automatically, currently as a compile time option. This can be used to disable polling via http requests. Is this of interest?

Also I have wrote more docs

Nice,

luc-github commented 1 year ago

The point is that the client announces what protocol version it wants to talk. If set via ESP800 then the backend tells itself via the client - this kind of defeats the purpose of having it.

Yeah seems I took problem upside down - no issue I plan to do several update before the week end - I can remove the subprotocol from [ESP800] and hardcode it in webUI - it will also simplify code on FW side

Here is one example.

Ho that is test server and seems ok - but I will update it as mentioned above - so it will be removed

2022-09-06T20:17:16

To be honest I just put - because I though will be easier to parse...but looking at code now I do not see the benefit... as I will update I can follow the format you mention - to be honest did not do any research on ISO format when I implemented it

I try to keep backwards compatibility when possible

Ok

This can be used to disable polling via http requests. Is this of interest?

Yes web UI parse all answers even no polling command to handle auto reporting, auto reporting is better if well managed on FW side , like if it does not pop in another running answer

terjeio commented 1 year ago

to be honest did not do any research on ISO format when I implemented it

Well, you were pretty close so you must have seen it before. I have created a c function for parsing the string that you may use if you want.

Yes web UI parse all answers even no polling command to handle auto reporting, auto reporting is better if well managed on FW side , like if it does not pop in another running answer

Yes, I saw that, even the $G response is handled when sent automatically (grblHAL can be configured to send it on changes).

From your commands memo it looks like json output is now the default, is this correct? Also many commands has pwd as a parameter. How/when is this intended to be used? When not logged in (no cookie present) and authentication is enabled?

luc-github commented 1 year ago

I have created a c function for parsing the string

Thank you I will have a look

it looks like json output is now the default

The webUI only use json yes, in ESP3D default output is plain text because if you type command in Serial/Telnet it for human reading and add json=yes to the command change output to json which is formated response for webUI, only ESP400 does not have plain text because I was lazy to do it as the plain text would not be really human readable anyway

many commands has pwd as a parameter.

Yes: the pwd=admin/user parameter is only for serial/telnet when authentication is enabled, http use the cookie once authenticated instead That said for serial ESP3D has 2 anothers settings :

terjeio commented 1 year ago

I've just committed an update for grblHAL with changes mostly for WebUI v3 support.

A question: is it per design that it is not possible to stream gcode files from flashfs? (flashfs is mapped to a littlefs mount directory in grblHAL).

luc-github commented 1 year ago

ESP400 now outputs all grblHAL settings, I emulate bitfields and axismasks via B type settings - this may change later when native support is available in the WebUI client

bit maks support is ready in code but not implemented in ESP400 I will add it soon

Flash filing system implemented by using littlefs

This the prefered choice also in ESP3D/ESP3DLib

I see that the ESP800 request does not add json=yes

all ESP command used by web UI need json=yes commands image You may use an old version of webUI if json=yes is no used

A question: is it per design that it is not possible to stream gcode files from flashfs? (flashfs is mapped to a littlefs mount directory in grblHAL).

Yes it is possible, I do not encourage this because of flash wearing and small size of flash, I can add it if you really want but I am confused I though you hide the flash partition GrblHal can have custom Flash it is possible, currently all targets use same but I can add custom one for GRBHal like for SD https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/FLASH-source.js#L24 https://github.com/luc-github/ESP3D-WEBUI/blob/3.0/src/targets/CNC/grblHAL/DIRECTSD-source.js The webUI is flexible ^_^

I will give a try to your latest version this week and check

luc-github commented 1 year ago

@terjeio I have do a fresh download of everything (https://github.com/grblHAL/ESP32#how-to-build-using-esp-idf-v43) and cannot compile :

no-stringop-truncation -MD -MT esp-idf/main/CMakeFiles/__idf_main.dir/webui/server.c.obj -MF esp-idf\main\CMakeFiles\__idf_main.dir\webui\server.c.obj.d -o esp-idf/main/CMakeFiles/__idf_main.dir/webui/server.c.obj   -c ../main/webui/server.c
../main/webui/server.c: In function 'wifi_scan_handler':
../main/webui/server.c:424:32: error: 'file' undeclared (first use in this function); did you mean 'pipe'?
                 vfs_puts(resp, file);
                                ^~~~
                                pipe
../main/webui/server.c:424:32: note: each undeclared identifier is reported only once for each function it appears in
../main/webui/server.c: In function 'wifi_login_receive_finished':
../main/webui/server.c:489:31: error: 'file' undeclared (first use in this function); did you mean 'pipe'?
     vfs_puts("Connecting...", file);
                               ^~~~
                               pipe
[1329/1334] Building C object esp-idf/main/CMakeFiles/__idf_main.dir/webui/commands_v3.c.obj
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

C:\Users\luc\Documents\GRBLHal\esp\esp-idf\ESP32>

Sorry I am out today and did not dig in issue much