collin80 / ESP32RET

CAN Reverse engineering tool for ESP32 based boards (specifically EVTV ESP32Due)
MIT License
178 stars 52 forks source link

ESP32 Keeps on crashing ("Core 1 panic'd") #25

Open embenix opened 10 months ago

embenix commented 10 months ago

First of all I am very pleased to say that I like this project. I am able to successfully compile and upload the project to ESP32 boards and test it together with SavvyCAN software to receive the frames. However, the frames kept on halting after a few seconds which had been a little frustrating. In the beginning I didn't know the exact problem but later with little more testing on Arduino's serial monitor I realized that the ESP32 boards keep on rebooting after a few seconds during the the scanning. I tested the code with different ESP32 boards including

I compiled the code using different versions of the esp boards in the board manager on Arduino and all of these boards showed more or less the same behavior. But the esp32 board version below 2.0.0 didn't compile because it didn't find <drivers/twai.h>.

Here is what it looks like when it reboots:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.

Core 1 register dump: PC : 0x400970ef PS : 0x00060033 A0 : 0x80096479 A1 : 0x3ffbf0e0
A2 : 0x00000022 A3 : 0x3ffc6324 A4 : 0x00000004 A5 : 0xb33fffff
A6 : 0x00000001 A7 : 0x00000001 A8 : 0x00000000 A9 : 0xb33fffff
A10 : 0x00000000 A11 : 0x00000003 A12 : 0x3ffe9f00 A13 : 0x3ffe9eb0
A14 : 0x00000001 A15 : 0x3ffc4390 SAR : 0x00000018 EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000032 LBEG : 0x400909a9 LEND : 0x400909cb LCOUNT : 0xffffffff

Backtrace:0x400970ec:0x3ffbf0e00x40096476:0x3ffbf100 0x400949a7:0x3ffbf120 0x400e398f:0x3ffbf150 0x40085319:0x3ffbf190 0x4000bfed:0x3ffe9e40 0x400973fd:0x3ffe9e50 0x40094791:0x3ffe9e70 0x400d6a1a:0x3ffe9eb0 0x400d6a71:0x3ffe9f00

and sometimes:

Backtrace:0x400970ec:0x3ffbf0e00x40096476:0x3ffbf100 0x400949a7:0x3ffbf120 0x400e2f73:0x3ffbf150 0x40085319:0x3ffbf190 0x400e36f9:0x3ffd2550 0x400dac8a:0x3ffd2590

Fun Fact: I tested the pre-compiled version of ESP32RET from Macchina found on this link, and it worked perfectly fine on ESP32 board (tested only on Huzzah32 so far).

Please help me fix this crashing problem. And if this still works fine for you, then I would like to know the exact versions of the libraries you used to compile the code i.e. Arduino, esp32 boards version in the board manager etc.

Thanks in advance.

collin80 commented 10 months ago

Yes, it works fine for me, I use it quite frequently so if there were a systematic problem I'd probably see it reasonably quickly. Still, I do have various reports of problems that people have when compiling it themselves. The first step is probably to try the "official" version I compiled: https://www.savvycan.com/ESP32RET_Updater.zip That was updated only a couple of weeks ago so it's up to date. If that works then the problem really is in the development environment on your side. If so we can try to figure out what part isn't getting set up properly. If it doesn't work with my compiled firmware then it must be something different with these other boards.

Also, if you download this: https://github.com/me-no-dev/EspExceptionDecoder then you can decode the backtraces and figure out where that is in code. I can't quite do that from your backtraces as it heavily depends on your library versions, the version of arduino-esp32, and the version of the Arduino IDE you're using. So, you have to decode your backtraces on your end. Then you can post what it says and that would help.

embenix commented 10 months ago

Thanks for the suggestions. As a matter of fact, I did use EspExceptionDecoder and it wasn't any big help at this point.

However, I have managed to fix the problem through the hard way. I tested all your commits together with the combination of "can_common" and "esp32_can" libraries. And it turns out that the "esp32_can" library had been causing the problem. And now I am using an older version of "esp32_can" library that you committed on Jan 18, 2022, found here: https://github.com/collin80/esp32_can/tree/0fb9878a77935d2995e094d196971009e0da3c48.

I highly appreciate your work and I would like to know if you have documented the GVRET CommProtocol?

uhi22 commented 9 months ago

Just cross-linking a similar crash topic, which is still under investigation: https://github.com/uhi22/wifican/blob/main/doc/crashlog.md#2023-10-05-crashlog-after-around-1740s-tracing-the-foccci-with-savvycan (It is not really related to ESP32RET, I created a minimal version of it, for easy debugging and to rule-out a lot of things.) The current analysis state is, that something wents wrong with the CAN receive queue in the espressif code.