I sometimes get read corruption after saving a preset multiple times quickly:
refreshing the page fixes the contents read from the slots file. usually the web UI automatically reloads the slot list with the correct list a split-second later. this time it didn't, so I was able to take a screenshot.
When the client visits /bin/slots.bin, replying with the slot name list is currently done by passing in a filesystem path to the server request->send(SPIFFS, "/slots.bin", "application/octet-stream"). The AsyncFileResponse opens the file, and the server incrementally sends the contents to the client, closing the file handle once finished.
While the server is still sending the file contents to the client, you can visit /slot/save, and tell the server to save the preset name list. SPIFFS doesn't block opening the file, but instead overwrites the same file the server is still reading from.
This causes the browser to see a corrupted slots.bin: slots race condition.bin. While the web server was transmitting the original file to the browser, midway through it starts reading from the beginning again (of the original or modified version, idk).
i don't know if all open handles to a filename have the same seek position. I don't know if this corrupts RAM, or the flash filesystem on-disk.
I discovered that every i2c register write (including saving a preset settings in loop()) and perhaps read too) calls yield(), which can launch wifi operations and callbacks when you don't expect it. I don't like this and want to prevent it from happening, but I'm not confident it's directly the cause of crashing and corruption.
i now believe the watchdog reboots are a side effect of RAM (or flash?) corruption.
i tried doing a flash backup (reads only!), and http://gbscontrol.local/spiffs/dir printed:
followed by a watchdog reboot (my serial terminal is corrupted so I don't know what it printed before reboot)
and now trying to backup again prints the same corrupted file list that isn't valid JSON. (protip: to clear corrupted files, you have to visit http://gbscontrol.local/spiffs/format before restoring a filesystem backup.)
it's possible (unconfirmed) that hardware watchdog resets, exceptions, file list and slots contents corruption results from a corrupted filesystem, originating from this confirmed "write a file being read" race condition. they may also arise from the ESPAsyncWebServer heap corruption above, heap OOM (i'm rebuilding with "new can abort" so OOM will print a message rather than invoking UB, and if I don't get messages this may not be the case), or a stack overflow (quite possible considering how horrifyingly long the call stack is).
Future
Ideally we'd do in some order:
Avoid asynchronous file reads from a web server, either by reading the whole file upfront or by not accepting further requests (and avoiding anything overwriting /slots.bin) until the request is finished.
Currently we handle "/slot/save" while reading the file. This is bad because it corrupts the file. But we can't block during "/slot/save" until the file is done reading, because you can't legally sleep in an AsyncWebServer callback.
I think we can't accept a web request while the HTTP "/slot/save" callback is writing into slots.bin. I'm not confident.
Verify the server won't do unwanted work in yield() (https://mongoose.ws/documentation/ should only do work when polled, not confident about ArduinoMongoose)
Setup the new server (could be difficult, unsure)
port the gbs callbacks to the wifi server (shouldn't be too much code to change)
port PersWiFiManager::setupWiFiHandlers() to the new server
should we replace PersWiFiManager::setupWiFiHandlers() with a static method returning a string, that you hook up to wifi yourself? though this makes it harder to return errors if we need to.
WebSockets.h is currently configured for ESPAsyncTCP, and does not support Mongoose. Mongoose has its own WebSockets support. We'll have to port that too.
possibly switch to platformio as a library manager and build system (sadly not debugger)
I had to delete all libraries in 3rdparty because they would get built and linked, but shouldn't be.
Additionally the resulting binaries take up more flash and RAM than Arduino. Unsure if this is caused by newer GCC or something else.
possibly switch to esp sdk v3 (newer GCC, better warnings)
possibly switch to littlefs
#include <LittleFS.h>, etc.
I do not plan to see this through to completion soon.
(draft, not fully edited)
Read corruption
I sometimes get read corruption after saving a preset multiple times quickly:
refreshing the page fixes the contents read from the slots file. usually the web UI automatically reloads the slot list with the correct list a split-second later. this time it didn't, so I was able to take a screenshot.
Diagnosis
the problem right now is:
/bin/slots.bin, replying with the slot name list is currently done by passing in a filesystem path to the serverrequest->send(SPIFFS, "/slots.bin", "application/octet-stream"). TheAsyncFileResponseopens the file, and the server incrementally sends the contents to the client, closing the file handle once finished./slot/save, and tell the server to save the preset name list. SPIFFS doesn't block opening the file, but instead overwrites the same file the server is still reading from.This causes the browser to see a corrupted slots.bin: slots race condition.bin. While the web server was transmitting the original file to the browser, midway through it starts reading from the beginning again (of the original or modified version, idk).
i don't know if all open handles to a filename have the same seek position. I don't know if this corrupts RAM, or the flash filesystem on-disk.
Crashes and persistent corruption
When saving presets, I've seen:
Exception crash log and stack trace
``` { /slot/save slotsBinaryFileRead = SPIFFS.open(SLOTS_FILE, "r") { File f(_impl->open(path, om, am), this) } File f, { f.setTimeCallback(timeCallback) } f.setTimeCallback(timeCallback) success slotsBinaryOutputFile = SPIFFS.open(SLOTS_FILE, "w") { File f(_impl->open(path, om, am), this) Exception (9): epc1=0x40000e5a epc2=0x00000000 epc3=0x00000000 excvaddr=0x40105857 depc=0x00000000 >>>stack>>> ctx: sys sp: 3fffdaf0 end: 3fffffb0 offset: 01a0 3fffdc90: 00000005 00000000 00000020 4010056c 3fffdca0: 4025e6cc 3ffef880 00000005 40101eb0 3fffdcb0: 3ffea565 4010515b 3ffedfa0 80000000 3fffdcc0: 00000020 3ffedfa0 00000000 00000000 3fffdcd0: 00007fff 18dfce7c 3ffeea38 40102c24 3fffdce0: 3ffeadf8 00000000 00000000 00000000 3fffdcf0: 00007fff 18dfce7c 401030e6 00000100 3fffdd00: 3ffeadf8 7fffffff 00002200 00000001 3fffdd10: 00000001 00004a08 00000000 00000000 3fffdd20: 3ffeadf8 00000000 00000000 18dfce7c 3fffdd30: 3ffeadec 2c9f0300 4000050c 3fffc278 3fffdd40: 40102e08 3fffc200 40228dcc 00000000 3fffdd50: 40004225 00000030 00000013 ffffffff 3fffdd60: 4000420c 00000000 3fffde40 08000000 3fffdd70: 60000200 00000002 02000000 02000000 3fffdd80: 02000000 00042035 00002035 60000200 3fffdd90: 3fff3c54 00000080 00318880 00000030 3fffdda0: 3ffeae04 00000000 00000000 00000100 3fffddb0: 00007fff 0133bc72 401030e6 00000100 3fffddc0: 3ffeae04 7fffffff 00002200 00000001 3fffddd0: 00000001 00000080 00000000 013399f5 3fffdde0: 3ffeae04 2c9f0300 4000050c 0133bc72 3fffddf0: 00000002 4000444e 4000050c 3fffc278 3fffde00: 00000000 4000422e 60000200 3fffdea4 3fffde10: 00000100 40004aa0 000000fc 00314c04 3fffde20: 00000100 3fffdea0 3fffc718 00314c04 3fffde30: 00000000 4000444e 40106578 00000004 3fffde40: 00000002 4000420c 60000200 3fff3c54 3fffde50: 00000100 40004ac0 00000100 00318800 3fffde60: 00000100 3fff3bd4 3fffc718 00318800 3fffde70: 00000100 00e8a101 40106578 00000100 3fffde80: 3fff3bd4 00318900 00000100 00318900 3fffde90: 3fffc718 3fff3bd4 00000001 4022b9f0 3fffdea0: 00318800 40225836 00318800 3fff3bd4 3fffdeb0: 00000000 4000422e 60000200 3fffdf54 3fffdec0: 00000100 40004aa0 000000fc 00318800 3fffded0: 00318900 00318900 00000100 4022b971 3fffdee0: 00000004 00e8a101 40106578 00000004 3fffdef0: 3fffdf50 00316b05 00000004 00316b05 3fffdf00: 3fffc718 3fffdf50 0000000a 00000000 3fffdf10: 00316b04 40225836 00316b04 3fffdf50 3fffdf20: 00000000 a0000000 00000000 0000001c 3fffdf30: 00004000 00e8a101 00004000 00000000 3fffdf40: 00316b05 00316b04 00000001 4022b9f0 3fffdf50: ffffff7e 3fffe043 60000200 3fff3cd4 3fffdf60: 00000100 40004b14 402359c9 00000004 3fffdf70: 3fffe0a0 00000004 3fffe0a0 40239fa2 3fffdf80: 3fffe000 3fff52a8 3fffdfb0 00000000 3fffdf90: 3fff3bd4 00316b00 00000100 40239ed8 3fffdfa0: 3fffe0a0 3fff52a8 3fffe000 40235af4 3fffdfb0: 00000004 40225836 00316a00 00000002 3fffdfc0: 00000000 400042db 3fffe0a0 40239fa2 3fffdfd0: 40004b31 3fff3f18 00000100 00318000 3fffdfe0: 401065c0 00316b00 00000100 3ffe9f6e 3fffdff0: 401005bd 3fff3f18 00318000 00318100 3fffe000: 00000000 00000004 00318000 40225861 3fffe010: 00000004 00000015 3fff1854 4022b884 3fffe020: 01004200 706d4501 20207974 00000000 3fffe030: 3fff3f18 00000100 0000082d 40101070 3fffe040: 00343032 706d4501 20207974 3fff3f04 3fffe050: 3fff3ddc 00318000 3fff1854 4022a2f8 3fffe060: 00000000 00000000 0000001f 4022fb26 3fffe070: 00000100 3fff3cd4 00000000 3fff3f18 3fffe080: 00000000 706d4501 20207974 00000057 3fffe090: 4022a2dc 00000030 00000013 00000100 3fffe0a0: 3fff1854 00018000 0000000c 4022d3bc 3fffe0b0: 00000000 00000080 00000000 00000f23 3fffe0c0: 00000001 00000f22 00000000 0000ffff 3fffe0d0: 3fff3cd4 00000001 00000020 00000100 3fffe0e0: 3fff3bd4 00000000 00000007 4022b89e 3fffe0f0: 00318800 3fff3f18 00000100 0000000c 3fffe100: 3fffe164 3fffe160 3fff1854 00000100 3fffe110: 3fff3bd4 00318800 3fff1854 4022a2dc 3fffe120: 3fff3ddc 00316a00 00316b00 4022fc2c 3fffe130: 3ffe9b12 00000007 3fff3ddc 4022b884 3fffe140: 00000000 400042db 3fff1854 0000007d 3fffe150: 3fff1854 3fff3bd4 00000188 4022ddc8 3fffe160: 00000007 0000000c 736a2f6e 3fff3f04 3fffe170: 3fff8002 00000001 00000000 00000002 3fffe180: 0000016a 00316b04 3fff1854 4022a2dc 3fffe190: 3fffe1d4 00316b04 3fff1854 4022fc2c 3fffe1a0: 3fff3df0 00000003 3fff3ddc 3fff397c 3fffe1b0: 3fff3bd4 0000ffff 3fff1854 4022e1c8 3fffe1c0: 3fffe1d0 00016b00 3fff1854 4022dc5a 3fffe1d0: 00000188 00003b7e 3fff1854 4022a2f8 3fffe1e0: 3fff3bd4 00008002 0000016a 00002000 3fffe1f0: 00000002 3fffe25e 00000000 00000000 3fffe200: 0000ffff 3fff397c 3fff1854 4022f36a 3fffe210: 00000000 ffffffff 3fffe228 00000020 3fffe220: 0000016a 3fff1854 016a397c 4022e2e6 3fffe230: 00000000 3fff3bd4 00000000 00000000 3fffe240: 00000000 00000000 0000016b 0000000a 3fffe250: 00000001 00000000 00000000 00000009 3fffe260: 00000100 0000000a 0000016a 00002000 3fffe270: 00000174 00017400 3fff1854 00000000 3fffe280: 00000000 00000016 3fff1854 4022c195 3fffe290: 3fff397c 016a0000 0a0d0a0d 4022b20d 3fffe2a0: 00000000 00000000 207d0029 4022b20d 3fffe2b0: 3ffea0f1 00000002 000008d3 3fff1854 3fffe2c0: 3fff184c 3ffe8bfa 3fffe338 4022aa09 3fffe2d0: 40225bec 3fff0934 3ffe9b12 40225fcd 3fffe2e0: 00000016 00000005 3fff0934 4022624c 3fffe2f0: 3ffe8bfa 3fff0ca8 3fff0934 3fff0934 3fffe300: 3ffe8bfa 3fff0ca8 3fffec70 40225b2e 3fffe310: 3ffe8cf8 402a42fd 3fff4f74 3fff0ca8 3fffe320: 40225bec 3fff0934 3ffe9b12 40225bf8 3fffe330: 40225bec 3fff0934 3ffe9b12 40225fcd 3fffe340: 00000002 00000005 3fff0934 4022624c 3fffe350: 3ffe8bfa 3fff478c 3fff0934 3fff0ca8 3fffe360: 3ffe8bfa 3fff478c 3fff0934 40203131 3fffe370: 20696957 70303639 20202000 20202020 3fffe380: 20202020 20202020 30000900 00010100 3fffe390: 20696957 70303834 20202000 20202020 3fffe3a0: 20202020 20202020 30002100 00010101 3fffe3b0: 74706d45 20202079 20202020 20202020 3fffe3c0: 20202020 20202020 00000000 01010002 3fffe3d0: 74706d45 20202079 20202020 20202020 3fffe3e0: 20202020 20202020 00000000 01010003 3fffe3f0: 74706d45 20202079 20202020 20202020 3fffe400: 20202020 20202020 00000000 01010004 3fffe410: 74706d45 20202079 20202020 20202020 3fffe420: 20202020 20202020 00000000 01010005 3fffe430: 74706d45 20202079 20202020 20202020 3fffe440: 20202020 20202020 00000000 01010006 3fffe450: 74706d45 20202079 20202020 20202020 3fffe460: 20202020 20202020 00000000 01010007 3fffe470: 74706d45 20202079 20202020 20202020 3fffe480: 20202020 20202020 00000000 01010008 3fffe490: 74706d45 20202079 20202020 20202020 3fffe4a0: 20202020 20202020 00000000 01010009 3fffe4b0: 74706d45 20202079 20202020 20202020 3fffe4c0: 20202020 20202020 00000000 0101000a 3fffe4d0: 74706d45 20202079 20202020 20202020 3fffe4e0: 20202020 20202020 00000000 0101000b 3fffe4f0: 74706d45 20202079 20202020 20202020 3fffe500: 20202020 20202020 00000000 0101000c 3fffe510: 74706d45 20202079 20202020 20202020 3fffe520: 20202020 20202020 00000000 0101000d 3fffe530: 74706d45 20202079 20202020 20202020 3fffe540: 20202020 20202020 00000000 0101000e 3fffe550: 74706d45 20202079 20202020 20202020 3fffe560: 20202020 20202020 00000000 0101000f 3fffe570: 74706d45 20202079 20202020 20202020 3fffe580: 20202020 20202020 00000000 01010010 3fffe590: 74706d45 20202079 20202020 20202020 3fffe5a0: 20202020 20202020 00000000 01010011 3fffe5b0: 74706d45 20202079 20202020 20202020 3fffe5c0: 20202020 20202020 00000000 01010012 3fffe5d0: 74706d45 20202079 20202020 20202020 3fffe5e0: 20202020 20202020 00000000 01010013 3fffe5f0: 74706d45 20202079 20202020 20202020 3fffe600: 20202020 20202020 00000000 01010014 3fffe610: 74706d45 20202079 20202020 20202020 3fffe620: 20202020 20202020 00000000 01010015 3fffe630: 74706d45 20202079 20202020 20202020 3fffe640: 20202020 20202020 00000000 01010016 3fffe650: 74706d45 20202079 20202020 20202020 3fffe660: 20202020 20202020 00000000 01010017 3fffe670: 74706d45 20202079 20202020 20202020 3fffe680: 20202020 20202020 00000000 01010018 3fffe690: 74706d45 20202079 20202020 20202020 3fffe6a0: 20202020 20202020 00000000 01010019 3fffe6b0: 74706d45 20202079 20202020 20202020 3fffe6c0: 20202020 20202020 00000000 0101001a 3fffe6d0: 74706d45 20202079 20202020 20202020 3fffe6e0: 20202020 20202020 00000000 0101001b 3fffe6f0: 74706d45 20202079 20202020 20202020 3fffe700: 20202020 20202020 00000000 0101001c 3fffe710: 74706d45 20202079 20202020 20202020 3fffe720: 20202020 20202020 00000000 0101001d 3fffe730: 74706d45 20202079 20202020 20202020 3fffe740: 20202020 20202020 00000000 0101001e 3fffe750: 74706d45 20202079 20202020 20202020 3fffe760: 20202020 20202020 00000000 0101001f 3fffe770: 74706d45 20202079 20202020 20202020 3fffe780: 20202020 20202020 00000000 01010020 3fffe790: 74706d45 20202079 20202020 20202020 3fffe7a0: 20202020 20202020 00000000 01010021 3fffe7b0: 74706d45 20202079 20202020 20202020 3fffe7c0: 20202020 20202020 00000000 01010022 3fffe7d0: 74706d45 20202079 20202020 20202020 3fffe7e0: 20202020 20202020 00000000 01010023 3fffe7f0: 74706d45 20202079 20202020 20202020 3fffe800: 20202020 20202020 00000000 01010024 3fffe810: 74706d45 20202079 20202020 20202020 3fffe820: 20202020 20202020 00000000 01010025 3fffe830: 74706d45 20202079 20202020 20202020 3fffe840: 20202020 20202020 00000000 01010026 3fffe850: 74706d45 20202079 20202020 20202020 3fffe860: 20202020 20202020 00000000 01010027 3fffe870: 74706d45 20202079 20202020 20202020 3fffe880: 20202020 20202020 00000000 01010028 3fffe890: 74706d45 20202079 20202020 20202020 3fffe8a0: 20202020 20202020 00000000 01010029 3fffe8b0: 74706d45 20202079 20202020 20202020 3fffe8c0: 20202020 20202020 00000000 0101002a 3fffe8d0: 74706d45 20202079 20202020 20202020 3fffe8e0: 20202020 20202020 00000000 0101002b 3fffe8f0: 74706d45 20202079 20202020 20202020 3fffe900: 20202020 20202020 00000000 0101002c 3fffe910: 74706d45 20202079 20202020 20202020 3fffe920: 20202020 20202020 00000000 0101002d 3fffe930: 74706d45 20202079 20202020 20202020 3fffe940: 20202020 20202020 00000000 0101002e 3fffe950: 74706d45 20202079 20202020 20202020 3fffe960: 20202020 20202020 00000000 0101002f 3fffe970: 74706d45 20202079 20202020 20202020 3fffe980: 20202020 20202020 00000000 01010030 3fffe990: 74706d45 20202079 20202020 20202020 3fffe9a0: 20202020 20202020 00000000 01010031 3fffe9b0: 74706d45 20202079 20202020 20202020 3fffe9c0: 20202020 20202020 00000000 01010032 3fffe9d0: 74706d45 20202079 20202020 20202020 3fffe9e0: 20202020 20202020 00000000 01010033 3fffe9f0: 74706d45 20202079 20202020 20202020 3fffea00: 20202020 20202020 00000000 01010034 3fffea10: 74706d45 20202079 20202020 20202020 3fffea20: 20202020 20202020 00000000 01010035 3fffea30: 74706d45 20202079 20202020 20202020 3fffea40: 20202020 20202020 00000000 01010036 3fffea50: 74706d45 20202079 20202020 20202020 3fffea60: 20202020 20202020 00000000 01010037 3fffea70: 74706d45 20202079 20202020 20202020 3fffea80: 20202020 20202020 00000000 01010038 3fffea90: 74706d45 20202079 20202020 20202020 3fffeaa0: 20202020 20202020 00000000 01010039 3fffeab0: 74706d45 20202079 20202020 20202020 3fffeac0: 20202020 20202020 00000000 0101003a 3fffead0: 74706d45 20202079 20202020 20202020 3fffeae0: 20202020 20202020 00000000 0101003b 3fffeaf0: 74706d45 20202079 20202020 20202020 3fffeb00: 20202020 20202020 00000000 0101003c 3fffeb10: 74706d45 20202079 20202020 20202020 3fffeb20: 20202020 20202020 00000000 0101003d 3fffeb30: 74706d45 20202079 20202020 20202020 3fffeb40: 20202020 20202020 00000000 0101003e 3fffeb50: 74706d45 20202079 20202020 20202020 3fffeb60: 20202020 20202020 00000000 0101003f 3fffeb70: 74706d45 20202079 20202020 20202020 3fffeb80: 20202020 20202020 00000000 01010040 3fffeb90: 74706d45 20202079 20202020 20202020 3fffeba0: 20202020 20202020 00000000 01010041 3fffebb0: 74706d45 20202079 20202020 20202020 3fffebc0: 20202020 20202020 00000000 01010042 3fffebd0: 74706d45 20202079 20202020 20202020 3fffebe0: 20202020 20202020 00000000 01010043 3fffebf0: 74706d45 20202079 20202020 20202020 3fffec00: 20202020 20202020 00000000 01010044 3fffec10: 74706d45 20202079 20202020 20202020 3fffec20: 20202020 20202020 00000000 01010045 3fffec30: 74706d45 20202079 20202020 20202020 3fffec40: 20202020 20202020 00000000 01010046 3fffec50: 74706d45 20202079 20202020 20202020 3fffec60: 20202020 20202020 00000000 01010047 3fffec70: 0000000b 00000538 00000538 00000000 3fffec80: 3fff2634 00000010 3fffed30 40227546 3fffec90: 3fff4d1d 00000018 40233858 00000000 3fffeca0: 000003e8 3ffe9b14 00000000 00000000 3fffecb0: 00000000 00000000 3fff0ca8 00000000 3fffecc0: 20202020 20202020 20202020 20202020 3fffecd0: 20202020 20202020 3fff4f00 20696957 3fffece0: 70303639 88ff4700 3fff0030 40227499 3fffecf0: 81ffed10 3fff2678 3fff4f14 402274b8 3fffed00: 00000000 3fffe370 3fffe380 40227499 3fffed10: 00000003 3fffed30 3fff2708 3ffe9b14 3fffed20: 3fff26f0 40201c5c 3fff26dc 402031e4 3fffed30: 3fff4cae 3fff26dc 3fff26dc 4021eecc 3fffed40: 00656461 3fff478c 3fff478c 4021cc68 3fffed50: 00594e00 70656500 806c612d 80006576 3fffed60: 3fff478c 3fff478c 3fff275c 4021f308 3fffed70: 800036f0 00000001 3fff47c0 00000002 3fffed80: 3fff4cae 3fff26dc 3fff478c 4021cd8d 3fffed90: 00002130 00000426 00000426 00000002 3fffeda0: 3fff4cae 00000002 3fff47c0 40228564 3fffedb0: 3fff4cae 00000423 00000423 00000002 3fffedc0: 3fff4cae 00000002 3fff478c 4021cfd8 3fffedd0: 00000001 3fff465e 00000020 401012fb 3fffede0: 00000000 4bc6a7f0 00065e10 3fff4b2c 3fffedf0: 3fff47c0 00000000 4bc6a7f0 00000000 3fffee00: 3fff0488 00000000 40100962 3fff4b2c 3fffee10: 00000000 3fffee60 3fff46a4 4021d031 3fffee20: 00000000 3fffee60 3fff46a4 402194b6 3fffee30: 3fff45e4 3fff478c 3fff46a4 3fff0f68 3fffee40: 3fff0000 00000000 00000020 3fff45e4 3fffee50: 3fff4b2c 00000000 3fff46a4 4021965e 3fffee60: 3fff3a5c 3fff3b9c 00000000 3fff0f68 3fffee70: 3fff10d8 3fff1188 3fff4b2c 3fff0f68 3fffee80: 3fff45e4 3fff0f4c 3fff0f4d 4023f4f0 3fffee90: 00003938 00000000 00000020 40101330 3fffeea0: 3fff10d8 f600a8c0 3fff176c 3fff1774 3fffeeb0: 00000000 00000000 00000020 3fff1188 3fffeec0: 3fff10d8 3fff4b4a 3fff4b2c 4024505a 3fffeed0: 00000014 3fff10d8 00000174 3fff1774 3fffeee0: 00000000 3fff16f4 00000020 401012fb 3fffeef0: 402577f8 3fff330c 3fff16f4 3fff1774 3fffef00: 00000008 3fff10d8 3fff4b2c 4023bfd1 3fffef10: 3fffdc80 3fff16f4 3fff330c 4023bdf4 3fffef20: 4025c512 3fff16f4 3fff330c 4025c523 3fffef30: 3fff4b3c 3fff4b2c 00000000 3fffdcc0 3fffef40: 4025741f 00000000 3fff330c 4025e2db 3fffef50: 40000f49 3fffdab0 3fffdab0 40000f49 3fffef60: 40000e19 40001878 00000002 3fffdad0 3fffef70: 3fffff10 aa55aa55 00000000 40105261 3fffef80: 40105267 00000002 3fffdad0 45470030 3fffef90: 4010000d 44005453 54454c45 55500045 3fffefa0: 40256300 3fffef3c 402562b9 3ffffed8 3fffefb0: 3fffffc0 00000000 00000000 feefeffe 3fffefc0: feefeffe feefeffe feefeffe feefeffe 3fffefd0: feefeffe feefeffe feefeffe feefeffe 3fffefe0: feefeffe feefeffe feefeffe feefeffe 3fffeff0: feefeffe feefeffe feefeffe feefeffe 3ffff000: feefeffe feefeffe feefeffe feefeffe 3ffff010: feefeffe feefeffe feefeffe feefeffe 3ffff020: feefeffe feefeffe feefeffe feefeffe 3ffff030: feefeffe feefeffe feefeffe feefeffe 3ffff040: feefeffe feefeffe feefeffe feefeffe 3ffff050: feefeffe feefeffe feefeffe feefeffe 3ffff060: feefeffe feefeffe feefeffe feefeffe 3ffff070: feefeffe feefeffe feefeffe feefeffe 3ffff080: feefeffe feefeffe feefeffe feefeffe 3ffff090: feefeffe feefeffe feefeffe feefeffe 3ffff0a0: feefeffe feefeffe feefeffe feefeffe 3ffff0b0: feefeffe feefeffe feefeffe feefeffe 3ffff0c0: feefeffe feefeffe feefeffe feefeffe 3ffff0d0: feefeffe feefeffe feefeffe feefeffe 3ffff0e0: feefeffe feefeffe feefeffe feefeffe 3ffff0f0: feefeffe feefeffe feefeffe feefeffe 3ffff100: feefeffe feefeffe feefeffe feefeffe 3ffff110: feefeffe feefeffe feefeffe feefeffe 3ffff120: feefeffe feefeffe feefeffe feefeffe 3ffff130: feefeffe feefeffe feefeffe feefeffe 3ffff140: feefeffe feefeffe feefeffe feefeffe 3ffff150: feefeffe feefeffe feefeffe feefeffe 3ffff160: feefeffe feefeffe feefeffe feefeffe 3ffff170: feefeffe feefeffe feefeffe feefeffe 3ffff180: feefeffe feefeffe feefeffe feefeffe 3ffff190: feefeffe feefeffe feefeffe feefeffe 3ffff1a0: feefeffe feefeffe feefeffe feefeffe 3ffff1b0: feefeffe feefeffe feefeffe feefeffe 3ffff1c0: feefeffe feefeffe feefeffe feefeffe 3ffff1d0: feefeffe feefeffe feefeffe feefeffe 3ffff1e0: feefeffe feefeffe feefeffe feefeffe 3ffff1f0: feefeffe feefeffe feefeffe feefeffe 3ffff200: feefeffe feefeffe feefeffe feefeffe 3ffff210: feefeffe feefeffe feefeffe feefeffe 3ffff220: feefeffe feefeffe feefeffe feefeffe 3ffff230: feefeffe feefeffe feefeffe feefeffe 3ffff240: feefeffe feefeffe feefeffe feefeffe 3ffff250: feefeffe feefeffe feefeffe feefeffe 3ffff260: feefeffe feefeffe feefeffe feefeffe 3ffff270: feefeffe feefeffe feefeffe feefeffe 3ffff280: feefeffe feefeffe feefeffe feefeffe 3ffff290: feefeffe feefeffe feefeffe feefeffe 3ffff2a0: feefeffe feefeffe feefeffe feefeffe 3ffff2b0: feefeffe feefeffe feefeffe feefeffe 3ffff2c0: feefeffe feefeffe feefeffe feefeffe 3ffff2d0: feefeffe feefeffe feefeffe feefeffe 3ffff2e0: feefeffe feefeffe feefeffe feefeffe 3ffff2f0: feefeffe feefeffe feefeffe feefeffe 3ffff300: feefeffe feefeffe feefeffe feefeffe 3ffff310: feefeffe feefeffe feefeffe feefeffe 3ffff320: feefeffe feefeffe feefeffe feefeffe 3ffff330: feefeffe feefeffe feefeffe feefeffe 3ffff340: feefeffe feefeffe feefeffe feefeffe 3ffff350: feefeffe feefeffe feefeffe feefeffe 3ffff360: feefeffe feefeffe feefeffe feefeffe 3ffff370: feefeffe feefeffe feefeffe feefeffe 3ffff380: feefeffe feefeffe feefeffe feefeffe 3ffff390: feefeffe feefeffe feefeffe feefeffe 3ffff3a0: feefeffe feefeffe feefeffe feefeffe 3ffff3b0: feefeffe feefeffe feefeffe feefeffe 3ffff3c0: feefeffe feefeffe feefeffe feefeffe 3ffff3d0: feefeffe feefeffe feefeffe feefeffe 3ffff3e0: feefeffe feefeffe feefeffe feefeffe 3ffff3f0: feefeffe feefeffe feefeffe feefeffe 3ffff400: feefeffe feefeffe feefeffe feefeffe 3ffff410: feefeffe feefeffe feefeffe feefeffe 3ffff420: feefeffe feefeffe feefeffe feefeffe 3ffff430: feefeffe feefeffe feefeffe feefeffe 3ffff440: feefeffe feefeffe feefeffe feefeffe 3ffff450: feefeffe feefeffe feefeffe feefeffe 3ffff460: feefeffe feefeffe feefeffe feefeffe 3ffff470: feefeffe feefeffe feefeffe feefeffe 3ffff480: feefeffe feefeffe feefeffe feefeffe 3ffff490: feefeffe feefeffe feefeffe feefeffe 3ffff4a0: feefeffe feefeffe feefeffe feefeffe 3ffff4b0: feefeffe feefeffe feefeffe feefeffe 3ffff4c0: feefeffe feefeffe feefeffe feefeffe 3ffff4d0: feefeffe feefeffe feefeffe feefeffe 3ffff4e0: feefeffe feefeffe feefeffe feefeffe 3ffff4f0: feefeffe feefeffe feefeffe feefeffe 3ffff500: feefeffe feefeffe feefeffe feefeffe 3ffff510: feefeffe feefeffe feefeffe feefeffe 3ffff520: feefeffe feefeffe feefeffe feefeffe 3ffff530: feefeffe feefeffe feefeffe feefeffe 3ffff540: feefeffe feefeffe feefeffe feefeffe 3ffff550: feefeffe feefeffe feefeffe feefeffe 3ffff560: feefeffe feefeffe feefeffe feefeffe 3ffff570: feefeffe feefeffe feefeffe feefeffe 3ffff580: feefeffe feefeffe feefeffe feefeffe 3ffff590: feefeffe feefeffe feefeffe feefeffe 3ffff5a0: feefeffe feefeffe feefeffe feefeffe 3ffff5b0: feefeffe feefeffe feefeffe feefeffe 3ffff5c0: feefeffe feefeffe feefeffe feefeffe 3ffff5d0: feefeffe feefeffe feefeffe feefeffe 3ffff5e0: feefeffe feefeffe feefeffe feefeffe 3ffff5f0: feefeffe feefeffe feefeffe feefeffe 3ffff600: feefeffe feefeffe feefeffe feefeffe 3ffff610: feefeffe feefeffe feefeffe feefeffe 3ffff620: feefeffe feefeffe feefeffe feefeffe 3ffff630: feefeffe feefeffe feefeffe feefeffe 3ffff640: feefeffe feefeffe feefeffe feefeffe 3ffff650: feefeffe feefeffe feefeffe feefeffe 3ffff660: feefeffe feefeffe feefeffe feefeffe 3ffff670: feefeffe feefeffe feefeffe feefeffe 3ffff680: feefeffe feefeffe feefeffe feefeffe 3ffff690: feefeffe feefeffe feefeffe feefeffe 3ffff6a0: feefeffe feefeffe feefeffe feefeffe 3ffff6b0: feefeffe feefeffe feefeffe feefeffe 3ffff6c0: feefeffe feefeffe feefeffe feefeffe 3ffff6d0: feefeffe feefeffe feefeffe feefeffe 3ffff6e0: feefeffe feefeffe feefeffe feefeffe 3ffff6f0: feefeffe feefeffe feefeffe feefeffe 3ffff700: feefeffe feefeffe feefeffe feefeffe 3ffff710: feefeffe feefeffe feefeffe feefeffe 3ffff720: feefeffe feefeffe feefeffe feefeffe 3ffff730: feefeffe feefeffe feefeffe feefeffe 3ffff740: feefeffe feefeffe feefeffe feefeffe 3ffff750: feefeffe feefeffe feefeffe feefeffe 3ffff760: feefeffe feefeffe feefeffe feefeffe 3ffff770: feefeffe feefeffe feefeffe feefeffe 3ffff780: feefeffe feefeffe feefeffe feefeffe 3ffff790: feefeffe feefeffe feefeffe feefeffe 3ffff7a0: feefeffe feefeffe feefeffe feefeffe 3ffff7b0: feefeffe feefeffe feefeffe feefeffe 3ffff7c0: 00000000 00000000 0000001f 4010056c 3ffff7d0: 3ffea565 4010515b 3ffedf78 40105a01 3ffff7e0: 3ffea565 4010515b 3ffedff0 40101eb0 3ffff7f0: 40102a47 3ffedff0 3ffeea38 40102c24 3ffff800: 00007fff 01393475 3ffeea38 40102c24 3ffff810: 3ffeae10 00000000 00000000 00000100 3ffff820: 00007fff 01393475 401030e6 00000100 3ffff830: 3ffeae10 7fffffff 00002200 00000001 3ffff840: 00000001 00004208 00000000 01348fba 3ffff850: 3ffeae10 00000000 00000020 01393475 3ffff860: 3ffeadf8 2c9f0300 4000050c 3fffc278 3ffff870: 40102e08 3fffc200 00000022 ffffffff 3ffff880: 4010612e 00000030 00000018 ffffffff 3ffff890: 40106484 00000001 3feffe00 00000100 3ffff8a0: 00eaa101 60000200 00020000 00000000 3ffff8b0: 00000002 4000444e 0000001f 4010056c 3ffff8c0: 00000000 4000422e 60000200 3ffff964 3ffff8d0: 00000100 40004aa0 000000fc 00317904 3ffff8e0: 00000100 3ffff960 3fffc718 00317904 3ffff8f0: 00000004 00e8a101 40106578 00000004 3ffff900: 3ffff960 00317905 00000004 00317905 3ffff910: 00000002 4000444e 000000fc 00317b04 3ffff920: 00000002 4000444e 60000200 3ffff9c4 3ffff930: 00000000 4000422e 60000200 3ffff9d4 3ffff940: 00000002 4000444e 000000fc 00318204 3ffff950: 00000000 4000422e 60000200 3ffff9f4 3ffff960: 00000100 40004aa0 000000fc 00316804 3ffff970: 00000100 3ffff9f0 3fffc718 00316804 3ffff980: 00000004 00e8a101 40106578 00000004 3ffff990: 3ffff9f0 00316805 00000004 00316805 3ffff9a0: 3fffc718 3ffff9f0 000000fc 00318604 3ffff9b0: 00316804 40225836 00316804 3ffff9f0 3ffff9c0: 00000004 00e8a101 40106578 00000004 3ffff9d0: 3ffffa30 00318605 00000004 00000000 3ffff9e0: 00316805 00316804 00000001 4022b9f0 3ffff9f0: ffffff7e 40225836 00318604 3ffffa30 3ffffa00: 00000000 4000422e 60000200 3ffffaa4 3ffffa10: 00000100 40004aa0 000000fc 00000000 3ffffa20: 00318605 00318604 00000001 4022b9f0 3ffffa30: fffffff8 00e8a101 40106578 00000004 3ffffa40: 00000002 4000444e 00000004 00316905 3ffffa50: 00000000 4000422e 60000200 3ffffb00 3ffffa60: 00000100 40004aa0 000000f8 00318708 3ffffa70: 00000100 3ffffaf0 3fffc718 00318708 3ffffa80: 00000010 00e8a101 40106578 00000010 3ffffa90: 00000000 4000444e 00000010 00318718 3ffffaa0: 00000000 400041bc 60000200 3fff3cd4 3ffffab0: 00000100 40004b14 00000100 00318600 3ffffac0: 00000100 3fff3bd4 3fffc718 00318700 3ffffad0: 00000100 00000000 40106578 00000100 3ffffae0: 3fff3bd4 00318700 00000100 00318700 3ffffaf0: 3fffc718 3fff3bd4 31313031 30603131 3ffffb00: 00318600 40225836 00318600 3fff3bd4 3ffffb10: 0a0d2c30 0a0d2c30 0a0d2c30 0a0d2c30 3ffffb20: 2c383231 35310a0d 0a0d2c37 00318600 3ffffb30: 00318700 00318700 00000100 4022b971 3ffffb40: 300a0d2c 340a0d2c 300a0d2c 300a0d2c 3ffffb50: 300a0d2c 310a0d2c 0a0d2c35 0a0d2c30 3ffffb60: 0a0d2c30 0a0d2c33 0a0d2c30 0a0d2c32 3ffffb70: 0a0d2c30 0d2c3734 0d2c300a 2c38350a 3ffffb80: 2c360a0d 36310a0d 0a0d2c30 0a0d2c30 3ffffb90: 0d2c3431 0d2c370a 0d2c360a 4022b884 3ffffba0: 00000100 3fff3cd4 ffffd8be 00000000 3ffffbb0: 3fff4254 00000100 4025857f 00000001 3ffffbc0: ffffffff 00000000 3ffea6b1 00000008 3ffffbd0: 402585ce 3ffedcd0 3fff132c 00000001 3ffffbe0: 00000002 00000000 00000020 4010056c 3ffffbf0: 00000005 00000005 00000002 40101eb0 3ffffc00: 3ffea562 4025768b 3ffedcd0 00000020 3ffffc10: 00000000 4024ac23 3ffef188 3fff132c 3ffffc20: 00000000 00000002 00000000 3ffedcd0 3ffffc30: 3fff4392 401057ab 3fff32e8 3fff0d24 3ffffc40: 3fff16f4 00000000 3fff3ddc 00000100 3ffffc50: 00000005 00000000 00000020 4010056c 3ffffc60: 3ffee6b0 00016800 00000005 40101eb0 3ffffc70: 3ffea565 4010515b 3ffedf50 00000000 3ffffc80: 40102a47 3ffedf50 3fff435c 4023bc42 3ffffc90: 00007fff 18df2cff 3ffeea38 40102c24 3ffffca0: 3ffeae1c 00000000 00000000 4010056c 3ffffcb0: 00007fff 18df2cff 401030e6 00000100 3ffffcc0: 3ffeae1c 7fffffff 00002200 00000001 3ffffcd0: 00000005 00000000 00000020 4010056c 3ffffce0: 3ffeae1c 18df1f51 00000005 40101eb0 3ffffcf0: 3ffea565 4010515b 3ffedf50 4010056c 3ffffd00: 40102a47 3ffedf50 3fffc228 40105a01 3ffffd10: 00007fff 18df6ba6 3ffeea38 40102c24 3ffffd20: 3ffeae1c 00000000 00000000 ffffffff 3ffffd30: 00007fff 18df6ba6 401030e6 00000100 3ffffd40: 3ffeae1c 7fffffff 00002200 00000001 3ffffd50: 00000001 00004a08 00000020 4010056c 3ffffd60: 3ffeae1c 3fffc200 00000002 18df6ba6 3ffffd70: 3ffeae10 2c9f0300 4000050c 3fffc278 3ffffd80: 40102e08 3fffc200 00000022 095a0455 3ffffd90: 4010064d 00000030 0000001c ffffffff 3ffffda0: 4022944c 00000003 00000005 60000318 3ffffdb0: c0036021 00000001 00000001 3fff0716 3ffffdc0: ffffffff 3fffc6fc 00000064 3fff0ac4 3ffffdd0: 00000000 60000314 00000001 00000030 3ffffde0: 3ffeae04 00000000 00000000 402293b5 3ffffdf0: 00007fff 2c9f0300 4000050c 3fffc278 3ffffe00: 40102e08 3fffc200 00000022 00000001 3ffffe10: 40215db8 00000030 0000001c ffffffff 3ffffe20: 40215db8 00000000 00000000 3fff0690 3ffffe30: c0036031 00000001 00000001 3fff0693 3ffffe40: ffffffff 3fffc6fc 00000064 3fff0680 3ffffe50: 00000017 3fffff50 0000004d 00000030 3ffffe60: 40204388 00000017 000000f0 3fff0695 3ffffe70: 60000314 00000001 3fff0ac4 00000001 3ffffe80: 3fff0ac4 00000001 3fff0ac4 402293b5 3ffffe90: 3fff0ac4 00000000 0000002a 3fff0716 3ffffea0: 60000314 00000001 3fff0ac4 4022931c 3ffffeb0: 00000001 3fff0716 3fff0ac4 40229674 3ffffec0: 00000000 00000000 00000001 4010056c 3ffffed0: 3fffff40 00000017 000003e8 00000001 3ffffee0: 3fffff50 00000001 402289fb 3fffefa0 3ffffef0: 000000f0 3fffff20 3fff0680 40228a39 3fffff00: 0000004d 3fffff30 00000000 40215c51 3fffff10: 00000001 3ffe84e8 3fff0680 40203e75 3fffff20: 00000001 3ffe84e8 00000005 3ffe84f0 3fffff30: 0000009d 00000017 0000001f 3ffe84f0 3fffff40: 00000001 3ffe84f4 0000000b 4020962c 3fffff50: 0000002a 3ffe84f4 00000001 40207f7a 3fffff60: 3fff0018 3ffe84f4 3fff0934 402137aa 3fffff70: 00000000 00000000 feefeffe feefeffe 3fffff80: 00000000 0000000a 00000001 4010056c 3fffff90: 3fffdad0 00000000 3fff0a40 3fff0a80 3fffffa0: 3fffdad0 00000000 3fff0a40 40228ae4 <<I discovered that every i2c register write (including saving a preset settings in loop()) and perhaps read too) calls yield(), which can launch wifi operations and callbacks when you don't expect it. I don't like this and want to prevent it from happening, but I'm not confident it's directly the cause of crashing and corruption.
i now believe the watchdog reboots are a side effect of RAM (or flash?) corruption. i tried doing a flash backup (reads only!), and http://gbscontrol.local/spiffs/dir printed:
followed by a watchdog reboot (my serial terminal is corrupted so I don't know what it printed before reboot) and now trying to backup again prints the same corrupted file list that isn't valid JSON. (protip: to clear corrupted files, you have to visit http://gbscontrol.local/spiffs/format before restoring a filesystem backup.)
Speculation
https://github.com/me-no-dev/ESPAsyncWebServer/issues/204 seems to indicate filesystem corruption with AsyncWebServer and SPIFFS.
https://github.com/me-no-dev/ESPAsyncWebServer/issues/324 seems to indicate that simultaneous HTTP requests causes ESPAsyncWebServer to corrupt memory. The issue has been closed by stalebot without any resolution.
it's possible (unconfirmed) that hardware watchdog resets, exceptions, file list and slots contents corruption results from a corrupted filesystem, originating from this confirmed "write a file being read" race condition. they may also arise from the ESPAsyncWebServer heap corruption above, heap OOM (i'm rebuilding with "new can abort" so OOM will print a message rather than invoking UB, and if I don't get messages this may not be the case), or a stack overflow (quite possible considering how horrifyingly long the call stack is).
Future
Ideally we'd do in some order:
PersWiFiManager::setupWiFiHandlers()to the new serverPersWiFiManager::setupWiFiHandlers()with a static method returning a string, that you hook up to wifi yourself? though this makes it harder to return errors if we need to.#include <LittleFS.h>, etc.I do not plan to see this through to completion soon.