search

pycom / pycom-micropython-sigfox

A fork of MicroPython with the ESP32 port customized to run on Pycom's IoT multi-network modules.
MIT License
199 stars 167 forks source link

LoPy4 keeps crashing/rebooting! #166

Open lambertvdw opened 6 years ago

lambertvdw commented 6 years ago

Using: sysname='LoPy4', nodename='LoPy4', release='1.17.3.b1', version='v1.8.6-849-83e2f7f on 2018-03-19', machine='LoPy4 with ESP32', lorawan='1.0.2', sigfox='1.0.1'

Have an application with the main thread and two service threads (see in some older issues similar problems as I have). Dispite the fact my main thread does have: while True: try: [ My app code]
except Exception as exc: pycom.rgbled(0x800000) # Mid red I get unhandled exceptions!

Application sometimes runs for day's but then starts to 'crash/reboot'. In case of a crash a reboot is acceptable (if device recovers recovers) but most of the time the LoPy keeps crashing over and over again. I see that the first Register dump is different from all the following! Also the core dump from first scrash is bigger that the following onces.

Output: [My program normal print statements] Guru Meditation Error: Core 1 panic'ed (LoadProhibited) Exception was unhandled. Register dump: PC : 0x400f8505 PS : 0x00060830 A0 : 0x800f85d8 A1 : 0x3ffe3550
A2 : 0xffffffff A3 : 0x00000000 A4 : 0x00000000 A5 : 0xffffffff
A6 : 0x00000001 A7 : 0x00017fa3 A8 : 0xffffffff A9 : 0xffffffc0
A10 : 0x00000000 A11 : 0x00000000 A12 : 0x00000010 A13 : 0x3face5d0
A14 : 0xffffffc0 A15 : 0x00000001 SAR : 0x0000001a EXCCAUSE: 0x0000001c
EXCVADDR: 0xffffffff LBEG : 0x4009b490 LEND : 0x4009b49b LCOUNT : 0x00000000

Backtrace: 0x400f8505:0x3ffe3550 0x400f85d5:0x3ffe3580 0x400f2ef3:0x3ffe35b0 0x400fbe2d:0x3ffe35d0 0x400f4c38:0x3ffe3670 0x400f1545:0x3ffe36b0 0x400fb95f:0x3ffe36d0 0x400dd5f1:0x3ffe3760

================= CORE DUMP START ================= vD0AABEAAABsAQAA 3Nj7PzDX+z/U2Ps/ MNf7P3DY+z/nyf+LdFL8P3RS/D/c2Ps/bFL8PwIAAABswvs/bML7P9zY+z8AAAAA ........ ........ [Lots of chars!!!!] ........ ........ AAAAAIDl/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAACM5f0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= Rebooting... Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a4 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x3ffd595c A7 : 0x00000060 A8 : 0x800840a2 A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008e9a4 A13 : 0x3ffdae40
A14 : 0x3ffd2934 A15 : 0x00000004 SAR : 0x00000013 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b490 LEND : 0x4009b49b LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x400836a1:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= 5BoAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA ........ ........ [Lots of chars!!!!] ........ ........ AAAAAEk2CEAAAAAAqNgIQABw+j8AAAAAAAAAAAAAAAD//z+zAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwDT9PyRP/D8AAAAA AAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1124) esp_core_dump: Skipped 1 tasks with bad TCB! E (1129) esp_core_dump: Crashed task has been skipped! Rebooting... Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a4 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x3ffd595c A7 : 0x00000060 A8 : 0x800840a2 A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008e9a4 A13 : 0x3ffdae40
A14 : 0x3ffd2934 A15 : 0x00000004 SAR : 0x00000013 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b490 LEND : 0x4009b49b LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x400836a1:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= 5BoAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA ........ ........ [Lots of chars!!!!] ........ ........ AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1124) esp_core_dump: Skipped 1 tasks with bad TCB! E (1129) esp_core_dump: Crashed task has been skipped! Rebooting... Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a4 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x3ffd595c A7 : 0x00000060 A8 : 0x800840a2 A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008e9a4 A13 : 0x3ffdae40
A14 : 0x3ffd2934 A15 : 0x00000004 SAR : 0x00000013 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b490 LEND : 0x4009b49b LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x400836a1:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= 5BoAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZ

The LoPy can't recover!!!!!

Please advice.

danspndl commented 6 years ago

Hi @lambertvdw 👋

Can you share some code examples, so I can try to reproduce this issue? Thanks!

lambertvdw commented 6 years ago

Sorry to say but sharing code is not possible.

I’m running a test now with the watchdog enabled.

Kind regards,

Lambert van de Wijdeven System Architect

+31 6 51338362 IoT Application creation Signify - Research Hight Tech Campus 7-1.C35 5656AE, Eindhoven, the Netherlands

[Signify logo]

[1]https://www.youtube.com/c/SignifyCompany [2] https://twitter.com/Signifycompany [3] https://www.pinterest.com/signifycompany/ [4] https://plus.google.com/+Signifycompany [5] https://www.facebook.com/PhilipsLighting

Our global brands are

[Philips logo right size]http://www.philips.com/lighting [interact logo right size] https://www.interact-lighting.com/en

The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.

From: Daniel Spindelbauer notifications@github.com Sent: Tuesday, September 4, 2018 2:48 PM To: pycom/pycom-micropython-sigfox pycom-micropython-sigfox@noreply.github.com Cc: Lambert van de Wijdeven lambert.van.de.wijdeven@signify.com; Mention mention@noreply.github.com Subject: Re: [pycom/pycom-micropython-sigfox] LoPy4 keeps crashing/rebooting! (#166)

Hi @lambertvdwhttps://github.com/lambertvdw 👋

Can you share some code examples, so I can try to reproduce this issue? Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/pycom/pycom-micropython-sigfox/issues/166#issuecomment-418353820, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AiVLwUxuCrEYrxpoJbF1AXcDycyJGcd8ks5uXnZxgaJpZM4UNc8h.

The information contained in this email may be confidential and/or legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this email is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original email.

glagnar commented 5 years ago

Has this been resolved ? The device crashes in a loop, even though I have erased all files / flashed the device.

lambertvdw commented 5 years ago

Hi Thomas,

Thanks for asking!

Indeed this issue is still relevant. However it doesn’t appear often it is a unresolvable issue when it happens.

Two weeks ago I made a new logging of this issue, as abstract;

_... Normal execution for days, including console output … Then the following output appears:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited) Register dump: ... ================= CORE DUMP START ================= ... ================= CORE DUMP END =================

The following block continues forever: Begin Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff8028,len:8
load:0x3fff8030,len:1728
load:0x4009fa00,len:0
load:0x4009fa00,len:14584
entry 0x400a059c
Guru Meditation Error: Core  1 panic'ed (InstrFetchProhibited)
. Exception was unhandled.

End_

More detailed output but with removal of the detailed dump bodies;

More detailed description:

Normal execution for days

Wait .............Guru Meditation Error: Core 1 panic'ed (LoadProhibited)

. Exception was unhandled. Register dump: PC : 0x400fcfa0 PS : 0x00060030 A0 : 0x80168bb1 A1 : 0x3ffd4ec0
A2 : 0x19474b16 A3 : 0x3ffd4f50 A4 : 0x3ffdb26c A5 : 0x00000001
A6 : 0x00000000 A7 : 0x00000000 A8 : 0x800fbd92 A9 : 0x3ffdac80
A10 : 0x00000001 A11 : 0x00000001 A12 : 0x3ffc6000 A13 : 0x3ffc6000
A14 : 0x3ffdade0 A15 : 0x3ffdad40 SAR : 0x0000001e EXCCAUSE: 0x0000001c
EXCVADDR: 0x19474b16 LBEG : 0x4009b288 LEND : 0x4009b2b6 LCOUNT : 0xffffffff

Backtrace: 0x400fcfa0:0x3ffd4ec0 0x40168bae:0x3ffd4f00 0x401610f2:0x3ffd4f30 0x40159bc0:0x3ffd4f50

================= CORE DUMP START ================= ZJEAAA8AAABsAQAA qNn7PwDX+z+g2fs/ ANf7P0DZ+z/uPgOVdFL8P3RS/D+o2fs/bFL8PwIAAAA4w/s/OMP7P6jZ+z8AAAAA FwAAAKTJ+z93aWZpACF+U9q0KkAuDOwAAAAAAKDZ+z8CAAAAIAAGABcAAAABAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA

... 790 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYPw/ AGD8PwAAAAAw5f0/ABQAABgf+z8AAAAAAAAAAAAAAAAAYPw/AAAAAAoAAAD///9/ AAAAAIDl/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAACM5f0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x00000001 A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008b302 A13 : 0x3ffc13d0
A14 : 0x3ffc1418 A15 : 0x00000001 SAR : 0x00000011 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= EBsAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

... 145 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwDT9PyRP/D8AAAAA AAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1148) esp_core_dump: Skipped 1 tasks with bad TCB! E (1153) esp_core_dump: Crashed task has been skipped! Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x3ffd595c A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008e980 A13 : 0x3ffdadc0
A14 : 0x3ffd2934 A15 : 0x00000004 SAR : 0x0000001e EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= UBsAAAkAAABsAQAA 0F39P1Bc/T/IXf0/ UFz9P2Bd/T/zvglETGP9P6hQ/D/QXf0/oFD8PxkAAAD8HmAAWsWfOtBd/T8AAAAA AAAAAMxZ/T9JRExFAGPN9HDuoIIqFucAAAAAAMhd/T8AAAAAIQAGAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

... 145 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALO29g== TDQIQES8CEAwAAYAfSYNgIA0/T+Ia/0/AAAAANRr/T8AAAAAiGv9PwAAAABEvAiA YDT9PwAAAAD/////lFL8PwAAAAAEGfw/IAAGAAAAAAClpaWlpaWlpQAAAAAAAAAA AAAAAEU2CEAAAAAAhNgIQHhu+j8AAAAAAAAAAAAAAAD//z+zAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwDT9PyRP/D8AAAAA AAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1156) esp_core_dump: Skipped 1 tasks with bad TCB! E (1161) esp_core_dump: Crashed task has been skipped! Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0
A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000
A6 : 0x00000001 A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070
A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008b302 A13 : 0x3ffc13d0
A14 : 0x3ffc1418 A15 : 0x00000001 SAR : 0x00000011 EXCCAUSE: 0x00000014
EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= EBsAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited)

I really need a solution for this issue before I can install it in the production environment!

Firmware used  Pycom MicroPython 1.18.1.r1 [v1.8.6-849-b0520f1] on 2018-08-29; LoPy4 with ESP32 WDT on boot  Enabled

I noticed in a normal boot the output is; ets Jun 8 2016 00:22:57

rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Starting the WDT on boot

But in case of the the contiuous reboot the output is ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled.

Zooming into this it seems that for whatever reason my application crashes with a message; << Normal application output>> Guru Meditation Error: Core 1 panic'ed (LoadProhibited) . Exception was unhandled. Then the LoPy4 tries to reboot but before it outputs the text ‘Starting the WDT on boot’ it outputs ‘Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited)’

Can you please give advise?

glagnar commented 5 years ago

Hi Thomas,

Thanks for asking!

Indeed this issue is still relevant. However it doesn’t appear often it is a unresolvable issue when it happens.

Two weeks ago I made a new logging of this issue, as abstract;

_... Normal execution for days, including console output … Then the following output appears:

Guru Meditation Error: Core 1 panic'ed (LoadProhibited) Register dump: ... ================= CORE DUMP START ================= ... ================= CORE DUMP END =================

The following block continues forever: Begin Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff8028,len:8
load:0x3fff8030,len:1728
load:0x4009fa00,len:0
load:0x4009fa00,len:14584
entry 0x400a059c
Guru Meditation Error: Core  1 panic'ed (InstrFetchProhibited)
. Exception was unhandled.

End_

More detailed output but with removal of the detailed dump bodies;

More detailed description:

Normal execution for days

Wait .............Guru Meditation Error: Core 1 panic'ed (LoadProhibited)

. Exception was unhandled. Register dump: PC : 0x400fcfa0 PS : 0x00060030 A0 : 0x80168bb1 A1 : 0x3ffd4ec0 A2 : 0x19474b16 A3 : 0x3ffd4f50 A4 : 0x3ffdb26c A5 : 0x00000001 A6 : 0x00000000 A7 : 0x00000000 A8 : 0x800fbd92 A9 : 0x3ffdac80 A10 : 0x00000001 A11 : 0x00000001 A12 : 0x3ffc6000 A13 : 0x3ffc6000 A14 : 0x3ffdade0 A15 : 0x3ffdad40 SAR : 0x0000001e EXCCAUSE: 0x0000001c EXCVADDR: 0x19474b16 LBEG : 0x4009b288 LEND : 0x4009b2b6 LCOUNT : 0xffffffff

Backtrace: 0x400fcfa0:0x3ffd4ec0 0x40168bae:0x3ffd4f00 0x401610f2:0x3ffd4f30 0x40159bc0:0x3ffd4f50

================= CORE DUMP START ================= ZJEAAA8AAABsAQAA qNn7PwDX+z+g2fs/ ANf7P0DZ+z/uPgOVdFL8P3RS/D+o2fs/bFL8PwIAAAA4w/s/OMP7P6jZ+z8AAAAA FwAAAKTJ+z93aWZpACF+U9q0KkAuDOwAAAAAAKDZ+z8CAAAAIAAGABcAAAABAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA

... 790 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYPw/ AGD8PwAAAAAw5f0/ABQAABgf+z8AAAAAAAAAAAAAAAAAYPw/AAAAAAoAAAD///9/ AAAAAIDl/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAACM5f0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0 A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000 A6 : 0x00000001 A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070 A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008b302 A13 : 0x3ffc13d0 A14 : 0x3ffc1418 A15 : 0x00000001 SAR : 0x00000011 EXCCAUSE: 0x00000014 EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= EBsAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

... 145 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwDT9PyRP/D8AAAAA AAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1148) esp_core_dump: Skipped 1 tasks with bad TCB! E (1153) esp_core_dump: Crashed task has been skipped! Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0 A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000 A6 : 0x3ffd595c A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070 A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008e980 A13 : 0x3ffdadc0 A14 : 0x3ffd2934 A15 : 0x00000004 SAR : 0x0000001e EXCCAUSE: 0x00000014 EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= UBsAAAkAAABsAQAA 0F39P1Bc/T/IXf0/ UFz9P2Bd/T/zvglETGP9P6hQ/D/QXf0/oFD8PxkAAAD8HmAAWsWfOtBd/T8AAAAA AAAAAMxZ/T9JRExFAGPN9HDuoIIqFucAAAAAAMhd/T8AAAAAIQAGAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

... 145 lines of core dump

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALO29g== TDQIQES8CEAwAAYAfSYNgIA0/T+Ia/0/AAAAANRr/T8AAAAAiGv9PwAAAABEvAiA YDT9PwAAAAD/////lFL8PwAAAAAEGfw/IAAGAAAAAAClpaWlpaWlpQAAAAAAAAAA AAAAAEU2CEAAAAAAhNgIQHhu+j8AAAAAAAAAAAAAAAD//z+zAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwDT9PyRP/D8AAAAA AAAAAAAAAAAAAAAAAAAAAP////8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAOA0/T8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAADsNP0/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAA ================= CORE DUMP END ================= E (1156) esp_core_dump: Skipped 1 tasks with bad TCB! E (1161) esp_core_dump: Crashed task has been skipped! Rebooting... ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled. Register dump: PC : 0x00000000 PS : 0x00060031 A0 : 0x400836a0 A1 : 0x3ffc13f0 A2 : 0x3ffc559c A3 : 0x00800000 A4 : 0x00000000 A5 : 0x00000000 A6 : 0x00000001 A7 : 0x00000060 A8 : 0x8008409e A9 : 0x00000070 A10 : 0x3ffd2944 A11 : 0x3ffd59c9 A12 : 0x8008b302 A13 : 0x3ffc13d0 A14 : 0x3ffc1418 A15 : 0x00000001 SAR : 0x00000011 EXCCAUSE: 0x00000014 EXCVADDR: 0x00000000 LBEG : 0x4009b46c LEND : 0x4009b477 LCOUNT : 0xffffffff

Backtrace: 0x00000000:0x3ffc13f0 0x4008369d:0x3ffc1410 0x4000bfed:0x00000000

================= CORE DUMP START ================= EBsAAAkAAABsAQAA 7Ff9P8BW/T/kV/0/ wFb9P4BX/T8AAAAAJFT8PwxR/D/sV/0/BFH8PxQAAAAAAAAAAAAAAOxX/T8AAAAA BQAAAOhP/T9TZXJ2ZXJzAAAAAAAAAAAAAQAAAORX/T8AAAAAAAAAAAUAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAADkaf0/TGr9P7Rq/T8AAAAAAAAAAAEAAAAAAAAA x1pAPwAAAABouglAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited)

I really need a solution for this issue before I can install it in the production environment!

Firmware used  Pycom MicroPython 1.18.1.r1 [v1.8.6-849-b0520f1] on 2018-08-29; LoPy4 with ESP32 WDT on boot  Enabled

I noticed in a normal boot the output is; ets Jun 8 2016 00:22:57

rst:0x7 (TG0WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Starting the WDT on boot

But in case of the the contiuous reboot the output is ets Jun 8 2016 00:22:57

rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff8028,len:8 load:0x3fff8030,len:1728 load:0x4009fa00,len:0 load:0x4009fa00,len:14584 entry 0x400a059c Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited) . Exception was unhandled.

Zooming into this it seems that for whatever reason my application crashes with a message; << Normal application output>> Guru Meditation Error: Core 1 panic'ed (LoadProhibited) . Exception was unhandled. Then the LoPy4 tries to reboot but before it outputs the text ‘Starting the WDT on boot’ it outputs ‘Guru Meditation Error: Core 1 panic'ed (InstrFetchProhibited)’

Can you please give advise?

I seem to have narrowed it down a bit. My specific error only occurs when the LoPy is attached to a single specifik PyTrack board. My other PyTrack boards work fine.

lambertvdw commented 5 years ago

Good to mention that I don't have a PyTrack board. The LoPy4 is connected to an Expansionboard V2.1.

giampiero7 commented 5 years ago

I witness this behaviour too, both on LoPy4 and WiPy. I'm trying to narrow down the root cause, but so far the behaviour seems pretty random and the tests take very long... On some occasions, after the core panic'ed error, the Pycom gets completely stuck and a manual reboot is needed. Anyone able to share known causes of core panic'ed errors?

amotl commented 4 years ago

Dear @lambertvdw, @Glagnar and @giampiero7,

if you feel lucky, you might want to try our custom build just released on [1]. More background about this is available through [2].

If we are lucky together, this will improve the stability significantly. If you will be still receiving the core dumps, I will be happy if you would share its content with us. Please just don't paste it into the comment itself but put it into a file which you will be able to upload by dragging it into the text area.

Please be aware that when upgrading to the current 1.20.1 development release these builds are based upon, you will have to erase your device completely before flashing in order to keep things straight. You will find respective references to this on the forum. Hint: Use pycom-fwtool-cli --port /dev/ttyUSB0 erase_all, see also [3].

Please also be aware that this procedure will also erase the LoRa MAC stored on the device. @robert-hh thankfully outlined the procedure to restore it appropriately:

  • If you still know the value the LoraMac had before, then you can follow the procedure explained at https://forum.pycom.io/topic/1272/lora-mac-ffffffffffffffff/21.
  • If you do not know it, you will have to use the Pycom updater to restore it:
    • First, use the Pycom updater (GUI version) to install a recent image (like pybytes 1.20.1) from the Internet. You do not have to use it, the installation process itself already restores both the LoRa MAC and the Sigfox ID.
    • Then, use the Pycom updater to flash the custom image. Do not erase the flash in between.

As it turned out to gain more robustness for others already [4,5], we will be happy to learn if this happens to you as well.

With kind regards, Andreas.

[1a] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/LoPy4-1.20.1.r1-0.6.0-vanilla-dragonfly.tar.gz [1b] https://packages.hiveeyes.org/hiveeyes/foss/pycom/vanilla/WiPy-1.20.1.r1-0.6.0-vanilla-dragonfly.tar.gz [2] https://community.hiveeyes.org/t/investigating-random-core-panics-on-pycom-esp32-devices/2480 [3] https://community.hiveeyes.org/t/installing-the-recent-pycom-firmware-1-20-1-r1-requires-erasing-the-flash-memory-completely/2688 [4] https://community.hiveeyes.org/t/testing-the-custom-dragonfly-builds-on-pycom-devices/2746 [5] https://github.com/pycom/pycom-micropython-sigfox/issues/361#issuecomment-553399627