runtime exception:28 crash on NodeMCU ESP8266

[marcussinus]

Environment
Here is the environment that I used:

• Microcontroller: NodeMCU ESP8266
• Core/runtime: ESP8266_nodemcuv2, core 3.0.2
• IDE: Arduino IDE latest

(1) ** ESP8266 crash**: ArduinoJson lib 7.0.2 used: Compiled is with no errors or warnings. JsonDocument doc is used in code. After ESP8266 is running the code for some seconds, exception:28 with CPU run crash happens.

ArduinoJson lib 6.21.5 used: Compiled is with no errors or warnings. DynamicJsonDocument doc(256) is used. ESP8266 runs error free.

(2) ** AsyncTelegram lib:** it works with ArduinoJson 6.21.5 but NOT with 7.0.2. Same code, compiled once with with v6 and once witvh v7. It looks you made a grater change in your ArduinoJson. Consequently, it is possible that more Libs are affected.

---

Part of code on ESP8266:

// -------------------------------------------------------
// za string
void pubClientAttrString(String key, String str, bool p) {
  DynamicJsonDocument doc(256);  // v 6; JsonDocument doc .. v7

  doc[key] = str;

  String output;
  serializeJson(doc, output);

  if (p)
    Serial << F(" --> Sending attribute:") << output << EOL;

  psClient.publish("v1/devices/me/attributes", output.c_str());
}
// -------------------------------------------------------
// s PubSubClient
bool PSreconnect(bool p) {
  if (psClient.connected())
    return true;
  else {
    if (p)
      Serial.print(F("[PSC] Attempting MQTT connection to ThingsBoard..."));
    // Attempt to connect (clientId, username, password)
    if (psClient.connect("Doma delavnica", TB_TOKEN, NULL)) {
      if (p)
        Serial.println(" ok done.\n");
      // Subscribing to receive RPC requests
      if (!psClient.subscribe("v1/devices/me/rpc/request/+"))
        Serial << F("[PSC]  *** RPC psClient.subscribe error! ***") << EOL;
      // Sending current status
      if (p)
        Serial.print("[PSC] Sending attributes ... ");
      pubClientAttrString("ResetCauseStr", lastResetCause, 1);
      return true;
    }
    else {
      if (p) {
        Serial.print("[PSC_FAIL] [state=");
        Serial.print(psClient.state());
      }
      return false;
    }
  }
}

[bblanchon]

Hi @marcussinus,

the code for some seconds [then] crash happens.

This sounds like a heap exhaustion.

exception:28

LoadProhibitedCause. This could be a null dereference, which would confirm the heap exhaustion.

Could you provide a stack trace? For example, you can use EspExceptionDecoder.

Please open another issue for the AsyncTelegram library issue and include the error in the issue description.

Best regards, Benoit

[marcussinus]

Hi, Regarding Telegram: there is no crash or exeptions, it just doen't send any msgs from board. And I didn't dig deeper, I just downgraded ArduinoJson, sorry.

ESP8266: I hope this helps: Exception:Error:28 -> LoadProhibited: CPU tried to load memory from a region which is protected against reads c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Variant/variantslot.hpp(44): error 0x4020143b:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\collection/CollectionImpl.hpp:17:::(inlined by) ArduinoJson: c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\collection/collectiondata.hpp:88:::(inlined by) ArduinoJson: c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Object/ObjectImpl.hpp(32): error 0x40203db9:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Object/ObjectImpl.hpp(32): error 0x40203d71:ArduinoJson c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\umm_malloc\/umm_malloc.cpp(688): error 0x40100b50:free (inlined by) free at c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\umm_malloc\/umm_malloc.cpp:667 c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Object/objectdata.hpp(55): error 0x40203df0:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Variant/variantrefbase.hpp(52): error 0x4021045f:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Variant/variantrefbase.hpp:64:::(inlined by) ArduinoJson: C:\Users\ms\Proj\Arduino_Sketchbook\libraries\AsyncTelegram2\src\/AsyncTelegram2.cpp:280:::(inlined by) AsyncTelegram2: c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/core_esp8266_main.cpp(238): error 0x40100178:ets_post ??:?:::0x40102114:pp_post ??:?:::0x4025b5a7:ppTxPkt ??:?:::0x40244b83:ieee80211_output_pbuf ??:?:::0x401059ef:wdt_feed ??:?:::0x4025c497:pp_attach ??:?:::0x4025c4e6:pp_attach c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\umm_malloc\/umm_malloc.cpp(912): error 0x40100b84:malloc c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/WString.cpp(246): error 0x40216547:String ??:?:::0x401053ab:lmacRxDone ??:?:::0x40103336:wDev_ProcessFiq c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Json/textformatter.hpp(125): error 0x40204394:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Json/textformatter.hpp(123): error 0x40204377:ArduinoJson c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/uart.cpp(547): error 0x40215e4f:uart_write c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/WString.cpp(192): error 0x402165db:String c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/WString.cpp(369): error 0x4021697c:String c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\serialization/countingdecorator.hpp(17): error 0x40204952:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\serialization\writers/arduinostringwriter.hpp(44): error 0x40201718:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Json/jsonserializer.hpp(132): error 0x40204e74:unsigned int ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Object/objectdata.hpp(35): error 0x402046e1:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Object/ObjectImpl.hpp:24:::(inlined by) ArduinoJson: c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/uart.cpp(545): error 0x40215e54:uart_write ??:?:::0x40269968:etharp_output c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/Print.cpp(108): error 0x40214aac:Print C:\Users\ms\Proj\Arduino_Sketchbook\my_tests\ESP8266_AsyncBot/ESP8266_AsyncBot.ino(463): error 0x402059fe:chkBotMsgs() c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\umm_malloc\/umm_malloc.cpp(642): error 0x40100802:umm_free_core c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\memory/VariantPoolImpl.hpp(22): error 0x402188c8:ArduinoJson c:\users\ms\proj\arduino_sketchbook\libraries\arduinojson\src\arduinojson\memory/variantpoollist.hpp:104:::(inlined by) ArduinoJson: C:\Users\ms\Proj\Arduino_Sketchbook\my_tests\ESP8266_AsyncBot/ESP8266_AsyncBot.ino(758): error 0x40206749:pubClientAttrInt(String, long, bool) c:\users\ms\appdata\local\arduino15\packages\esp8266\hardware\esp8266\3.1.2\libraries\esp8266wifi\src\include/clientcontext.h(480): error 0x4020b31a:ClientContext C:\Users\ms\Proj\Arduino_Sketchbook\my_tests\ESP8266_AsyncBot/ESP8266_AsyncBot.ino(1529): error 0x40208ae4:loop c:\users\ms\appdata\local\arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.1.0-gcc10.3-e5f9fec\xtensa-lx106-elf\include\c++\10.3.0\bits/std_function.h(289): error 0x40218790:std c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/core_esp8266_main.cpp(258): error 0x40213768:loop_wrapper() c:\Users\ms\AppData\Local\Arduino15\packages\esp8266\hardware\esp8266\3.1.2\cores\esp8266\/cont.S(81): error 0x40100e1d:cont_wrapper

[bblanchon]

Unfortunately, the stack isn't very helpful. Could you create an MCVE that doesn't involve MQTT?

[marcussinus]

Well, MCVE without MQTT (like your examples) do work ok. So it is probably somehow connectect to it. But, the bottom line is: v6 works ok, v7 does not (with the same MQTT lib and settings). You can also see first error from stack report: \arduino_sketchbook\libraries\arduinojson\src\ArduinoJson/Variant/variantslot.hpp(44): error 0x4020143b:ArduinoJson

[bblanchon]

But, the bottom line is: v6 works ok, v7 does not (with the same MQTT lib and settings).

I understand it looks bad for ArduinoJson 7, but it doesn't necessarily mean there is a bug in v7. It could also come from a bug in the program that only reveals under special conditions, for example, when the heap is exhausted. Still, if the bug is in the library, there must be a way to reproduce it without the help of a third-party library.

Anyway, if you want to help me fix this problem, you must give me a complete reproduction code; the shorter, the better. The stack trace mentions an ESP8266_AsyncBot.ino of at least 758 lines; this is way too long. Can you reduce it to a 100-line repro? Please provide all the instructions so I can run the program here.

[marcussinus]

Ok, I did squeeze it, kind of. It has more than 100 lines, but do not be afraid, just start (at the end) of setup(). Attached code crashes there. It is a "downgraded" code, so some non-meaning variables/code might be present, just ignore them.

Code crashes with ArduinoJson 7.0.2, but not with 6.21.5.

Arduino IDE is latest. Also all other libraries are up-to-date. I tested it on NodeMCU 8266 (esp8266_nodemcu and esp8266_nodemcuv2), and on ESP8266-07 with esp8266_generic.

You will need to have a Telegram account and set up a bot. If you have it - super, if not - it is easy to set one. Then insert correct BOT_TOKEN and BOT_ID. No account - no crash ;)

WiFi setup should not be a problem.

Thingsboard MQTT .. it crashes on my compile even without correct server data, so, no connection shall be needed to produce a crash. But, if you need/want, you can setup your own free account on their server for MQTT.

Hope it helps! ESP8266_Json_test.zip

[willmmiles]

Comment from the peanut gallery: I've been looking at upgrading ArduinoJson from v6 to v7 in WLED, another large project, and almost immediately started running in to strange crashes on my ESP8266s that turned out to be stack exhaustion. And not in a small way, either - serializing the large config document seemed to use almost 2kb more stack than v6, a real challenge when the stack size is only 4k to begin with. The config document is not nested particularly deep either, 4 levels at most. I came looking here to see if someone else had reported the same issue.

I'm still investigating -- I've worked around it for debugging by increasing the stack size, I'm going to look at the compiled output next and see if there's some issues with the compiler regarding variable scope or inlining that are making things so large.

[bblanchon]

@willmmiles, please open a dedicated issue. We'll continue the conversation there.

[bblanchon]

@marcussinus, thank you very much for reducing the code size. However, I can see many lines that don't play any role in this bug. Please remove every line that is not essential to the reproduction of the bug.

[bblanchon]

Line 274:

psClient.setBufferSize(512); 

Then, line 123:

char pload[256];
memcpy(pload, payload_in, length);

Therefore, every message longer than 256 characters long corrupts the stack. This explains why the program crashes and why the stack trace was corrupted. This bug was silent with ArduinoJson 6, but that was just luck.

Solution: get rid of the pload buffer and pass payload_in directly to deserializeJson():

deserializeJson(doc, payload_in, length);

See: How to use ArduinoJson with PubSubClient?

[marcussinus]

Hi, version 7.03 works like a charm 😀 (sorry for my late reply, I was abroad)

Thank you!