Open SparkyDan555 opened 1 year ago
The problem is likely to be related to auto-reporting. We have some work in progress to resolve this. In the meantime, if you get the WebUI from https://github.com/MitchBradley/ESP3D-WEBUI/blob/revamp/index.html.gz and use the preferences dialog to set the auto-report interval to 0, it might help.
Thanks for the reply. I have tried your suggestion but sadly it doesn't seem to help. I have set the auto-report interval to 0 which makes "Auto" unselectable. I set the reporting to "None" but this setting appears to be reset on boot.
Logs:
[MSG:INFO: FluidNC v3.7.1 (main-e99a8abb-dirty)]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.4]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:config.yaml]
[MSG:INFO: Machine TMC2130 XY Laser]
[MSG:INFO: Board FluidNC Laser (Daniel White)]
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]
[MSG:INFO: SD Card cs_pin:gpio.5 detect:NO_PIN freq:8000000]
[MSG:INFO: Stepping:RMT Pulse:2us Dsbl Delay:0us Dir Delay:1us Idle Delay:250ms]
[MSG:INFO: User Digital Output:1 on Pin:gpio.27]
[MSG:INFO: User Digital Output:2 on Pin:gpio.2]
[MSG:INFO: User Digital Output:3 on Pin:gpio.21]
[MSG:INFO: Axis count 3]
[MSG:INFO: Shared stepper disable gpio.13]
[MSG:INFO: Axis X (0.000,500.000)]
[MSG:INFO: Motor0]
[MSG:INFO: tmc_2130 Step:gpio.14 Dir:gpio.12 CS:gpio.16 Disable:NO_PIN Index:-1 R:0.110]
[MSG:INFO: X Neg Limit gpio.36]
[MSG:INFO: X All Limit gpio.34:low]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Y (0.000,410.000)]
[MSG:INFO: Motor0]
[MSG:INFO: tmc_2130 Step:gpio.25 Dir:gpio.26 CS:gpio.17 Disable:NO_PIN Index:-1 R:0.110]
[MSG:INFO: Y Neg Limit gpio.39]
[MSG:INFO: Y All Limit gpio.35:low]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Z (-1000.000,0.000)]
[MSG:INFO: Motor0]
[MSG:INFO: X Axis driver test passed]
[MSG:INFO: Y Axis driver test passed]
[MSG:INFO: safety_door_pin gpio.32:pu]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Laser Ena:NO_PIN Out:gpio.22 Freq:5000Hz Period:8191]
[MSG:INFO: Using spindle Laser]
[MSG:INFO: Connecting to STA SSID:########]
[MSG:INFO: Connecting.]
[MSG:INFO: Connecting..]
[MSG:INFO: Connected - IP is 10.2.1.91]
[MSG:INFO: WiFi on]
[MSG:INFO: Start mDNS with hostname:http://fluidnc.local/]
[MSG:INFO: SSDP Started]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]
Grbl 3.7 [FluidNC v3.7.1 (main-e99a8abb-dirty) (wifi+bt) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]
[MSG:WARN: Low memory: 12960 bytes]
[MSG:WARN: Low memory: 12768 bytes]
[MSG:WARN: Low memory: 12640 bytes]
[MSG:WARN: Low memory: 12240 bytes]
[MSG:WARN: Low memory: 11792 bytes]
[MSG:WARN: Low memory: 11660 bytes]
[MSG:WARN: Low memory: 11532 bytes]
[MSG:WARN: Low memory: 11404 bytes]
[MSG:WARN: Low memory: 11276 bytes]
[MSG:WARN: Low memory: 11148 bytes]
[MSG:WARN: Low memory: 11020 bytes]
[MSG:WARN: Low memory: 9620 bytes]
[MSG:WARN: Low memory: 9500 bytes]
[MSG:WARN: Low memory: 9372 bytes]
[MSG:WARN: Low memory: 9252 bytes]
[MSG:WARN: Low memory: 9132 bytes]
[MSG:WARN: Low memory: 9124 bytes]
[MSG:WARN: Low memory: 9092 bytes]
[MSG:WARN: Low memory: 7580 bytes]
[MSG:WARN: Low memory: 7420 bytes]
[MSG:WARN: Low memory: 7292 bytes]
[MSG:WARN: Low memory: 7164 bytes]
[MSG:WARN: Low memory: 7036 bytes]
[MSG:WARN: Low memory: 5344 bytes]
[MSG:WARN: Low memory: 5216 bytes]
[MSG:WARN: Low memory: 5088 bytes]
[MSG:WARN: Low memory: 5004 bytes]
[MSG:WARN: Low memory: 4780 bytes]
[MSG:WARN: Low memory: 4648 bytes]
[MSG:WARN: Low memory: 4520 bytes]
[MSG:WARN: Low memory: 4392 bytes]
[MSG:WARN: Low memory: 4264 bytes]
[MSG:WARN: Low memory: 4136 bytes]
[MSG:WARN: Low memory: 3984 bytes]
[MSG:WARN: Low memory: 3860 bytes]
[MSG:WARN: Low memory: 3732 bytes]
[MSG:WARN: Low memory: 3336 bytes]
[MSG:WARN: Low memory: 3156 bytes]
[MSG:WARN: Low memory: 2896 bytes]
[MSG:WARN: Low memory: 2776 bytes]
[MSG:WARN: Low memory: 2648 bytes]
[MSG:WARN: Low memory: 2592 bytes]
[MSG:WARN: Low memory: 2584 bytes]
[MSG:WARN: Low memory: 2556 bytes]
[MSG:WARN: Low memory: 2168 bytes]
If you set both the status report interval and the auto-report interval to 0, None will be used.
I made a prerelease with my latest attempt to fix this problem. Please try it out. https://github.com/bdring/FluidNC/releases/tag/v3.7.2-pre1
Unfortunately this hasn't helped.
Logs:
[MSG:INFO: FluidNC v3.0.x (noGit)]
[MSG:INFO: Compiled with ESP32 SDK:v4.4.4]
[MSG:INFO: Local filesystem type is spiffs]
[MSG:INFO: Configuration file:config.yaml]
[MSG:INFO: Machine TMC2130 XY Laser]
[MSG:INFO: Board FluidNC Laser (Daniel White)]
[MSG:INFO: SPI SCK:gpio.18 MOSI:gpio.23 MISO:gpio.19]
[MSG:INFO: SD Card cs_pin:gpio.5 detect:NO_PIN freq:8000000]
[MSG:INFO: Stepping:RMT Pulse:2us Dsbl Delay:0us Dir Delay:1us Idle Delay:250ms]
[MSG:INFO: User Digital Output:1 on Pin:gpio.27]
[MSG:INFO: User Digital Output:2 on Pin:gpio.2]
[MSG:INFO: User Digital Output:3 on Pin:gpio.21]
[MSG:INFO: Axis count 3]
[MSG:INFO: Shared stepper disable gpio.13]
[MSG:INFO: Axis X (0.000,500.000)]
[MSG:INFO: Motor0]
[MSG:INFO: tmc_2130 Step:gpio.14 Dir:gpio.12 CS:gpio.16 Disable:NO_PIN Index:-1 R:0.110]
[MSG:INFO: X Neg Limit gpio.36]
[MSG:INFO: X All Limit gpio.34:low]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Y (0.000,410.000)]
[MSG:INFO: Motor0]
[MSG:INFO: tmc_2130 Step:gpio.25 Dir:gpio.26 CS:gpio.17 Disable:NO_PIN Index:-1 R:0.110]
[MSG:INFO: Y Neg Limit gpio.39]
[MSG:INFO: Y All Limit gpio.35:low]
[MSG:INFO: Motor1]
[MSG:INFO: Axis Z (-1000.000,0.000)]
[MSG:INFO: Motor0]
[MSG:INFO: X Axis driver test passed]
[MSG:INFO: Y Axis driver test passed]
[MSG:INFO: safety_door_pin gpio.32:pu]
[MSG:INFO: Kinematic system: Cartesian]
[MSG:INFO: Laser Ena:NO_PIN Out:gpio.22 Freq:5000Hz Period:8191]
[MSG:INFO: Using spindle Laser]
[MSG:INFO: Connecting to STA SSID:########]
[MSG:INFO: Connecting.]
[MSG:INFO: Connecting..]
[MSG:INFO: Connected - IP is 10.2.1.91]
[MSG:INFO: WiFi on]
[MSG:INFO: Start mDNS with hostname:http://fluidnc.local/]
[MSG:INFO: SSDP Started]
[MSG:INFO: HTTP started on port 80]
[MSG:INFO: Telnet started on port 23]
Grbl 3.0 [FluidNC v3.0.x (noGit) (wifi+bt) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]
[MSG:WARN: Low memory: 14992 bytes]
[MSG:WARN: Low memory: 14744 bytes]
[MSG:WARN: Low memory: 12268 bytes]
[MSG:WARN: Low memory: 12068 bytes]
[MSG:WARN: Low memory: 11940 bytes]
[MSG:WARN: Low memory: 11880 bytes]
[MSG:WARN: Low memory: 11552 bytes]
[MSG:WARN: Low memory: 11504 bytes]
[MSG:WARN: Low memory: 11376 bytes]
[MSG:WARN: Low memory: 11248 bytes]
[MSG:WARN: Low memory: 11120 bytes]
[MSG:WARN: Low memory: 11072 bytes]
[MSG:WARN: Low memory: 10996 bytes]
[MSG:WARN: Low memory: 10864 bytes]
[MSG:WARN: Low memory: 10736 bytes]
[MSG:WARN: Low memory: 10608 bytes]
[MSG:WARN: Low memory: 10420 bytes]
[MSG:WARN: Low memory: 10292 bytes]
[MSG:WARN: Low memory: 10240 bytes]
[MSG:WARN: Low memory: 9876 bytes]
[MSG:WARN: Low memory: 8692 bytes]
[MSG:WARN: Low memory: 8680 bytes]
[MSG:INFO: Channel auto report interval set to 50 ms]
[MSG:WARN: Low memory: 7532 bytes]
[MSG:WARN: Low memory: 7372 bytes]
[MSG:WARN: Low memory: 7220 bytes]
[MSG:WARN: Low memory: 6200 bytes]
[MSG:WARN: Low memory: 6008 bytes]
Try removing the SD card to see if the memory that is used to read it could be related.
Also try issuing the $Heap command to see how much memory is available after things settle down.
And set $message/level=debug in case that shows something interesting.
I'm using the standard build (from release's package 3.7.1) on an MKS DLC32 board, and I got the same warning messages. Low memory and counting downwards. For me, setting report to none (in web UI) has solved it. At least the messages stop. After a reboot, it starts again (because the web UI seems to set it back to 50ms). So, I suspect your suggestion with auto-reporting is correct.
The warning message was added in 3.7.1. Whatever is using the memory has probably not changed; the difference is that it is now being reported while previously it went unnoticed.
I am looking for what could be using the memory. This is a tricky problem because the likely culprit is something deep inside the network stack, beneath two layers of third-party code. It is also likely to depend on details of what else is happening on the wifi network, thus very difficult to replicate in another environment or to reproduce at will.
I think I have a way to repro the problem. If I flood-ping the FluidNC controller with sudo ping -l 96 -s 1420 -f 192.168.68.139
from a Linux system, the Low memory warning happens immediately, dropping to below 3K in less than one second, and hanging the ESP32 network stack so it stops responding to pings.
I have tried variations of that command with different preload and packet sizes, leading me to think that the network stack is being presented with a lot of packets in a short time. It has to allocate memory for each until it is dealt with, then the memory can be returned back to the heap. $HEAP shows that, indeed, the memory is being properly returned, but there is an interval when a lot of memory is being used up by network packets that have not yet been processed.
A similar thing could happen on a network that is not being subjected to a ping flood. There are some network services that use broadcasting, where every node must receive the packet and do something with it. Other services or agents might do address-scanning, periodically sending an inquiry to every possible address. I know that some mesh routers do something like that; in fact we had to add a filter in the FluidNC HTTP server to detect and discard inquiries from Google WiFI mesh routers which use them to automatically discover when a new router has been added to the network.
"Fixing" this is going to be very difficult, since it is happening deep inside the SDK code which we normally try not to touch - because modifying it requires a lot of work to re-plumb things and creates a difficult maintenance problem going forward.
The conditions on the WiFi network are not under our control. We can't stop another computer from sending arbitrary amounts of random stuff that the ESP32 network stack will have to look at before it can release the memory it used to hold the incoming packets.
We can make the "WARN" message go away; for example we could move it to DEBUG level so most people will not see it. But that could just hide an underlying problem. If the memory gets too low, even temporarily, FluidNC might crash or hang, without any warning or indication why.
One mitigation might be to put FluidNC behind an extra router that would filter traffic so FluidNC does not have to see all the chitchat that is happening on the main network.
Also note that this test was performed without WebUI. The starting point was the wifi version immediately after reboot, with no network connections active, just FluidTerm monitoring the serial port. WebUI will probably make it die a bit sooner due to the modest amount of memory that is necessary to maintain the WebUI websocket connection - but if you push the ESP32 network stack hard enough, it will run out of memory with or without WebUI.
The same thing happens with not only 3.7.1, but also some development versions that try to be extra-careful about memory use and flow control for websockets. It boils down to "if the wifi network presents enough packets in a short time to the underlying ESP32 network stack, it will allocate memory for them until it runs out".
I found a way to configure the memory usage of the WiFi module. Try building from the latest version of the WebSocketsNoStall branch https://github.com/bdring/FluidNC/tree/WebSocketsNoStall . Use $heap periodically to monitor heap usage. Hopefully the low-water mark will stay above the warning threshold.
Thanks for the reply and apologies for the delayed response. I have tried the WebSocketsNoStall branch but unfortunately cannot report good news. I still get the warning messages and the captive portal 'File index.html. gz is missing, please upload it' page is unresponsive at times. When I try to upload the files, the upload freezes and the log reports:
[MSG:WARN: Low memory: 720 bytes]
[MSG:INFO: Upload cancelled]
[MSG:WARN: Low memory: 620 bytes]
This is with the wifibt build (which is the reason I am using a 16 MB board). So I tried just a wifi build and the problem is actually worse. The captive portal page barely loads at all, although no warning messages are seen in the console? Neither build was stable enough to upload index.html.gz and reach the main home page.
I am not sure that we can ever stabilize the wifibt build because BT uses so much memory. Doubling the FLASH does not help with RAM.
Can you upload the index.html.gz with FluidTerm?
I can upload it via FluidTerm but still can't access the main page due to stability.
Also, just to note that the wifibt build FluidNC v3.6.8 (main-33576ba2) (wifi+bt) is responsive and stable.
The most invasive change that occurred since 3.6.8 was the change from using Arduino String to C++ std::string . To see if that change triggered your problem, please test with builds from #c444b46 (before) and #ddb5628a (after).
The precompiled Bluetooth library in the Arduino Espressif32 framework uses a lot of RAM. The WebSocketsNoStall branch has code to release the Bluetooth RAM if $Bluetooth/Enable=Off .
https://github.com/MitchBradley/BTLib is a setup to compile an alternative version of libbt.a that omits unnecessary features like BLE, audio, and whatnot, thus saving a significant amount of RAM even when Bluetooth is enabled.
I spent more that two full days working on this, benefiting no other users that I know of. Please consider becoming a sponsor.
The change from using Arduino String to C++ std::string doesn't appear to impact MSG:WARN: Low memory:
Grbl 3.6 [FluidNC v3.6.7 (HEAD-c444b463-dirty) (wifi+bt) '$' for help]
[MSG:ERR: '$H'|'$X' to unlock]
[MSG:INFO: WiFi Disconnected]
Cannot connect to WebUI
Grbl 3.6 [FluidNC v3.6.7 (HEAD-ddb5628a-dirty) (wifi+bt) '$' for help]
[MSG:ERR: '$H'|'$X' to unlock]
[MSG:ERR: mount_to_vfs failed code 0x0x101]
[MSG:ERR: mount_to_vfs failed code 0x0x101]
[MSG:ERR: mount_to_vfs failed code 0x0x101]
Can connect to WebUI
The latest WebSocketsNoStall branch still gives warnings:
Grbl 3 [FluidNC v3.7,2-pre3 (WebSocketsNoStall-9da58e98-dirty) (wifi+bt) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]
[MSG:WARN: Low memory: 5816 bytes]
and with the new libbt.a...
Grbl 3 [FluidNC v3.7,2-pre3 (WebSocketsNoStall-9da58e98-dirty) (wifi+bt) '$' for help]
[MSG:INFO: '$H'|'$X' to unlock]
[MSG:WARN: Low memory: 5804 bytes]
Do you have both bluetooth and wifi enabled at the same time? You need to turn one of them off, say with $bluetooth/enable=off or $wifi/mode=off
I have made sure that bluetooth is disabled which helped with the WebSocketsNoStall build but not the main build, I still see warning messages.
There is no point testing the main build, as it does not have any of the numerous fixes that I have done recently. Do you still see warning messages with the WebSocketsNoStall build? Your message above was unclear.
No warning messages with WebSocketsNoStall after $bluetooth/enable=off
Controller Board
FluidNC Pen Laser Controller (SPI)
Machine Description
Feungsake 4040 Laser Engraver with 4W PWM Laser Module
Input Circuits
No response
Configuration file
Startup Messages
User Interface Software
FluidTerm
What happened?
I have built a 16MB version of FluidNC for a 16MB ESP32-DevKit with the following lines in platform.ini:
This does work but when connecting to the web interface, the console is filled with
MSG:WARN: Low memory
. I have a previous build up to commit 468fee9 with the same changes in platformio.ini which does not have this issue.Other Information
No response