Closed AdiXPS closed 2 years ago
@AdiXPS could you tell us which modem are you using? Also, are you seeing the first crash with just the code you've shared or was there anything else running on ESP32 at the same time?
As for the first issue, null dereference in the internal UART structures, I couldn't think of anything else but memory corruption. Is this happening randomly or could it be somehow related to the second issue? (like one occurs after another?)
As for the second problem, it might be possible that +++
command won't exit the data mode, this is typically defined in the specification of the device used (and for our supported devices, the procedure is to repeat the sequence after one second)
Hi @david-cermak, Sure, I am using sim7070g, mostly like sim800. There is no other task is running other than connect and disconnect.
As for the first issue, null dereference in the internal UART structures, I couldn't think of anything else but memory corruption. Is this happening randomly or could it be somehow related to the second issue? (like one occurs after another?)
Its randomly. I notice that its crashing on Core 1? not sure why though.. moreover, based on my observation, its not related as its rare..
As for the second problem, it might be possible that +++ command won't exit the data mode, this is typically defined in the specification of the device used (and for our supported devices, the procedure is to repeat the sequence after one second)
Its the same as sim800. i've added 1s delay before initiating ath command (based on last change you've proposed in another issue). But that didn't prevent the issue from happening again.. I notice one thing that modem in exiting PPP is still sometimes sending data by UART to esp even when confirming for +++..
"10/30/2021 11:33:35 PM",CONNECT 150000000
"10/30/2021 11:33:35 PM",~ÿ}#À!}!í} }9}"}&} } } } }#}%Â#}%}%}&Ø}+‘¹}'}"}(}" ˆÿ}#À!}"}!} }4}"}&} } } } }%}&tUl+}'}"}(}"7õÿ}#À!}!î} }8}"}&} } } } }#}$À#}%}&Ø}+‘¹}'}"}(}"À}:ÿ}#À!}+ï} }(Ø}+‘¹qBÀ# ý0€!n 1Ö€!
"10/30/2021 11:33:36 PM", -ø0€!o 팀!
"10/30/2021 11:33:36 PM",D’TTë7ƒTë9æn~~€!
"10/30/2021 11:33:36 PM",D’TTë7ƒTë9æ_„~
"10/30/2021 11:33:40 PM",OK <=============this is when esp sends +++, module sends OK but esp keeps getting data afterwards..
"10/30/2021 11:33:43 PM",~!E s³ì@ é+ä6ÌÞO <==== this is the famous ~!E when code crash..
"10/30/2021 11:33:43 PM",D’T»äêE3-_€ @?ë
"10/30/2021 11:33:43 PM",@`Åñ!ÿç "ûÞzêFçÈ°rúß“Ü={k¹Å!Ðá¤êz©]Z¼ëŠí ¡#<©€¯•Y‹›Fèõ8ü%©B‚~~!E s•@ êÃ;6ÌÞO
"10/30/2021 11:33:43 PM",D’T»å@T „àAJ€ 9põ
"10/30/2021 11:33:43 PM",™Æ{( £ ".KþZ0^#}]ãÝtPV#HÌMÁÜe‡5šT8½ê ØwG«+ÙéßÍ6H×@-ÜDÛóQ~
Its happening randomly
@AdiXPS
I notice one thing that modem in exiting PPP is still sometimes sending data by UART to esp even when confirming for +++..
This is weird, I've never seen anything like that with any device. Is the OK
response surrounded by <CR><LF>
? Could you please share binary hexdump?
Are you sure, you don't use any kind of virtual terminal feature/CMUX mode? Or isn't it that your device is somehow configured to multiplex terminals?
Anyway, I'll look into it and try to reproduce the issue, at least the first one. For the second one, we'd probably have to purchase your device sim7070g
This is weird, I've never seen anything like that with any device.
Yup, I totally agree with you.
Is the
OK
response surrounded by<CR><LF>
?
Yes it is.
Could you please share binary hexdump?
I appreciate to get your advice regarding proper way to get the hexdump.
Are you sure, you don't use any kind of virtual terminal feature/CMUX mode? Or isn't it that your device is somehow configured to multiplex terminals?
I've double check, mux is not enabled.
Anyway, I'll look into it and try to reproduce the issue, at least the first one. For the second one, we'd probably have to purchase your device
sim7070g
Thanks.. from what I can see that there are other issues raised here facing same issue, but no one actually tried to check the data coming from modem. I always notice ~!E
in others facing the issue.. might it be a buffer issue, as data is coming even after exiting +++
?
@AdiXPS
I appreciate to get your advice regarding proper way to get the hexdump.
You can use ESP_LOG_BUFFER_HEXDUMP()
macro from IDF's log component, it's used for example here:
IDF version 4.3
Are you sure this was the correct version?
config.dte_buffer_size = CONFIG_EXAMPLE_MODEM_UART_RX_BUFFER_SIZE / 2;
This line from your code suggests it might have been v4.4
? (dte_buffer_size
was added in 4.4)
I've been testing the while loop on release/v4.3
on SIM7600
and no issues, so far. I'd guess that the condition of incorrect exit of PPP mode might have caused some memory corruption. I'll focus on this and try to simulate this condition somehow.
@AdiXPS Thanks for reporting, would you please help share if any updates for the issue? Thanks.
@AdiXPS Thanks for reporting, would you please help share if any updates for the issue? Thanks.
Still there is no update, same issue is occurring. Actually just today I got new sim7070g, I'll de-solder the one I'm using and replace it with new one just in case. I'll rerun it again with hexdump and I'll post the update within two days God willing.
@david-cermak @Alvin1Zhang I am still facing the same issue with different SIM7070G module. Different PCB also. @david-cermak Regarding hexdump may I know where exactly I need to place it, to help more in the investigation e.g. at the end of esp_handle_uart_data() function?
@david-cermak , @Alvin1Zhang We'll, seems that the issue is the module 7070G/7080 have backdoor after full analysis. The modem keeps sending data even after exiting data mode, I have also requested to get the latest firmware from simcom (just to eliminate any M-in-M kind of issues) but its even worse now. Its getting certificates and all kind of alarming stuff:
it tries to get fake cert. for *.piojm.tech and this brings me to conclude that all ~!E famous text received is when simcom tries to establish malicious connections..
"11/29/2021 10:49:10 PM",]߱-¹IW:~ "11/29/2021 10:49:11 PM",OK
"11/29/2021 10:49:15 PM",OK <===ATH confirmation "11/29/2021 10:49:15 PM",~!E xߴ@ 3ڴ¤Zi "11/29/2021 10:49:15 PM",o»ʕس¼؈א < J F/· ʔi
=춒ͽ]½핉 "11/29/2021 10:49:15 PM",Ę À0 ÿ # h2] Y V û0÷0ߠ:BG8ӅÀw¯ ª¹R0 "11/29/2021 10:49:15 PM", *H÷ "11/29/2021 10:49:15 PM", 0Y10 UUS10U "11/29/2021 10:49:15 PM",DigiCert Inc1301U*RapidSSL TLS DV RSA Mixed SHA256 2020 CA-10 "11/29/2021 10:49:15 PM",200810000000Z "11/29/2021 10:49:15 PM",220811120000Z010U*.piojm.tech0"0 "11/29/2021 10:49:15 PM", *H÷ "11/29/2021 10:49:15 PM", 0 "11/29/2021 10:49:15 PM", ²qf6põ|5RB1<½;ވ¯䌌´¯r!ÀN$Vb✇Uڏ>ù^´¤Zűü¯gͮ褔 ۟ݞøx&»}^p¸褼½ފ"11/29/2021 10:49:16 PM",+|8#5ɅώˮKհ "11/29/2021 10:49:16 PM",^2¡»õ¤ûy¯W7kѨ¨ᔢ°e1u?H1.§(ٴک[¨2sJ䆤k5püU㣃��┸_5*2«גѷ?r¦(PA$¿٦¸ꌉO¹ùuO$ΌV$upŬP3讖¤»
?b_±¬9HĖ £û0÷0U#0¤徼y䰣m.)4#Xܵ10U˶He�]ýmX噘0#U0.piojm.tech "11/29/2021 10:49:16 PM",piojm.tech0Uÿ 0U%0++0LU E0C07 `Hýl00(+https://www.digicert.com/CPS0g0 +y0w0$+0http://ocsp.digicert.com0O+0Chttp://cacerts.digicert.com/RapidSSLTLSDVRSAMixedSHA2562020CA-1.crt0 U0 0}^ "11/29/2021 10:49:16 PM",+ֹnjh u )y¾𞹹!c¥w得}]` "11/29/2021 10:49:16 PM",øùM]&\%]DŽ süߠ F0D (L8ö@²¹j߅K®¡A[Vþµڢ Ժ¯ Q7´®QIY¹xµ髼E˦3趥l´䠷 "EEYU$V?¡/ࣦcÀK]ƃ\n⏂ sý H0F! Р¤;͠YþĞYM¹;'A®緌zȣjcOy! Rf䒄¹´<ùÀõ ͟&Bxӿ띠¯N^Ơv Q£°õýyVm¸7x¤z̛'˷B "11/29/2021 10:49:16 PM",þԋ堠sý[ G0E ={é慦%:䟙9��A-f/ПU¦4! ¢n³l t3꘠e�EZ)z»ù¶TIڈd0 "11/29/2021 10:49:17 PM", *H÷ "11/29/2021 10:49:17 PM",þ5~~!E 4=+ 7¡û%"11/29/2021 10:49:17 PM",o»˝µƕ "11/29/2021 10:49:17 PM",º7 "11/29/2021 10:49:17 PM",E½z'-¹IW@
I am not interested in this aspect (backdoor or not), how can we ignore such coming data?
Hi @AdiXPS
The famous ~!..
is just an ascii representation of start of frame of PPP and an IP protocol id. This is what you've already pointed out above, that the device didn't exit the PPP session, but is still talking in PPP back. Couldn't be the backdoor data just public certs of some connection, e.g. forcefully retried after the hangup?
I couldn't reproduce the issue with the devices supported in IDF, which makes me think it has to be device specific, something to do with 7070G/7080
modems.
how can we ignore such coming data?
I think you can try to reset the device, either using a reset command or pulling the reset line directly. Or you can try to use CMUX mode, but it depends on your usecase. What's the reason for switching to command mode? Do you need to use AT interface or close the network connection?
Hi @david-cermak
Couldn't be the backdoor data just public certs of some connection, e.g. forcefully retried after the hangup?
Hi @david-cermak
Couldn't be the backdoor data just public certs of some connection, e.g. forcefully retried after the hangup?
I highly doubt this, as we got confirmation from the module 'OK' (mentioned earlier in the logs) for exiting data and hangup, but the module keeps sending and receiving data. As long as we got ATH => OK this should kill any connection, regardless if its public cert or not. Moreover, even if its public cert, its no where mentioned all of their documentation regarding this behavior/activity. btw the domain cert that we've found in the module is considered as one of the blacklisted DNSs. I am reporting the findings here for anyone will use this module.
What's the reason for switching to command mode? Do you need to use AT interface or close the network connection?
I need to close the connection and get the lon,lat by AT command. and do battery readings.. but as long as we keep getting such data uart will be unuasable
I need to close the connection and get the lon,lat by AT command.
You can use CMUX mode for that and keep the network/PPP still on.
Hi @AdiXPS
Is there any update from your side? This issue seems to be related to specifically your devices of 7070G/7080
family, that are not supported in IDF examples. Could you please close the issue, then? (note, that it's also okay to raise a feature request to support your devices, if you're thinking of that, please do so in the esp-protocols
repo)
Thanks for reporting, will close due to short of feedback, feel free to reopen with more updates. Thanks.
Environment
Problem Description
I've tried @david-cermak example, basiclly I found two main issues after leave it running for a couble of hours:
1-after awhile, I'am getting the famous "GGuru Mediation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled." One more thing here, I notice that its showing Core 1 while I dont have any task running on it..?
`Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump: PC : 0x400d52e5 PS : 0x00050031 A0 : 0x400826f8 A1 : 0x3ffb0dc0 0x400d52e5: uart_ll_get_intsts_mask at C:/esp4/esp-idf/components/hal/esp32/include/hal/uart_ll.h:170 (inlined by) uart_rx_intr_handler_default at C:/esp4/esp-idf/components/driver/uart.c:746
0x400826f8: _xt_lowint1 at C:/esp4/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105
A2 : 0x3ffb7760 A3 : 0x3ffb0aa0 A4 : 0x00000091 A5 : 0x8ca96391
A6 : 0x00000910 A7 : 0x00000000 A8 : 0x00000001 A9 : 0x3ffb0d50 A10 : 0x00000000 A11 : 0x00000000 A12 : 0x80103e4e A13 : 0x3ffb86c0
A14 : 0x3ffb2f6c A15 : 0x3ffb86e0 SAR : 0x0000001a EXCCAUSE: 0x0000001c
EXCVADDR: 0x00000008 LBEG : 0x00000000 LEND : 0x00000000 LCOUNT : 0x00000000
Backtrace:0x400d52e2:0x3ffb0dc0 0x400826f5:0x3ffb0e10 0x40118563:0x3ffb8710 0x400d845b:0x3ffb8730 0x40088505:0x3ffb8750 0x4008a525:0x3ffb8770 0x400d52e2: uart_ll_get_intsts_mask at C:/esp4/esp-idf/components/hal/esp32/include/hal/uart_ll.h:170 (inlined by) uart_rx_intr_handler_default at C:/esp4/esp-idf/components/driver/uart.c:746
0x400826f5: _xt_lowint1 at C:/esp4/esp-idf/components/freertos/port/xtensa/xtensa_vectors.S:1105
0x40118563: cpu_ll_waiti at C:/esp4/esp-idf/components/hal/esp32/include/hal/cpu_ll.h:183 (inlined by) esp_pm_impl_waiti at C:/esp4/esp-idf/components/esp_pm/pm_impl.c:827
0x400d845b: esp_vApplicationIdleHook at C:/esp4/esp-idf/components/esp_common/src/freertos_hooks.c:63
0x40088505: prvIdleTask at C:/esp4/esp-idf/components/freertos/tasks.c:3839 (discriminator 1)
0x4008a525: vPortTaskWrapper at C:/esp4/esp-idf/components/freertos/port/xtensa/port.c:168
ELF file SHA256: d843f1542d225f73`
2-Mostly the crash is happening when stopping PPP only if there is existing connection. If for any reason the disconnection happen from host (ISP; no IP EVENT) its very stable.
[0;32mI (511128) esp-modem: entering esp_modem_stop_ppp...[0m [0;31mE (512058) esp-modem: esp_dte_handle_line(124): handle line failed[0m [0;33mW (512058) pppos_example: Unknow line received: ~!E[0m Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled.
Always getting Unknow line received: ~!E.
When I monitor the modem I am getting this transmitted:
"10/30/2021 3:50:38 PM",CONNECT 150000000 "10/30/2021 3:50:38 PM",~ÿ}#À!}!¦} }9}"}&} } } } }#}%Â#}%}%}&Öcº÷}'}"}(}"£~~ÿ}#À!}"}!} }4}"}&} } } } }%}&M§T}$}'}"}(}"³ø~~ÿ}#À!}!§} }8}"}&} } } } }#}$À#}%}&Öcº÷}'}"}(}"í×~~ÿ}#À!}+¨} }(Öcº÷CC~~À# ý0~~€!< jà~~€! "10/30/2021 3:50:39 PM", -ø0~~€!= ¶º~~€! "10/30/2021 3:50:39 PM",™ mTë7ƒTë9æÄG~~€! "10/30/2021 3:50:39 PM",™ mTë7ƒTë9æõÙ~~!E ;å@ UÄ "10/30/2021 3:50:43 PM",E "10/30/2021 3:50:43 PM",™ m»b
µÇª€ }]y"10/30/2021 3:50:43 PM",¼}^€bV_9 x(~ "10/30/2021 3:50:43 PM",OK
"10/30/2021 3:50:47 PM",OK "10/30/2021 3:50:59 PM",~!E (®²@ 3Ú¶13‚. "10/30/2021 3:50:59 PM",™ m»âpK.hì¶ÑP s‘
“~~!E ;æ@ UÃ "10/30/2021 3:51:23 PM",E "10/30/2021 3:51:23 PM",™ m»b
µÇª€ áw"10/30/2021 3:51:23 PM",¼cV_9 µ~~!E 0 @ sIa13‚.`
seems the esp sending +++ and ath but the modem keep sending garbage? or it didnt actually exit data mode?
here is sample from modem when there is no issues..:
`"10/30/2021 5:05:32 PM",CONNECT 150000000 "10/30/2021 5:05:32 PM",~ÿ}#À!}!}1} }9}"}&} } } } }#}%Â#}%}%}&Ö¨N½}'}"}(}"°}#
ÿ}#À!}"}!} }4}"}&} } } } }%}&×É£Õ}'}"}(}"_Dÿ}#À!}!}2} }8}"}&} } } } }#}$À#}%}&Ö¨N½}'}"}(}"¡¿ÿ}#À!}+}3} }(Ö¨N½œ°À# ý0€!Ø ªŠ€! "10/30/2021 5:05:34 PM", -ø0€!Ù vЀ! "10/30/2021 5:05:34 PM",ANTë7ƒTë9毥~~€! "10/30/2021 5:05:34 PM",ANTë7ƒTë9æž;~ "10/30/2021 5:05:38 PM",OK"10/30/2021 5:05:41 PM",OK
"10/30/2021 5:05:52 PM",OK`
Expected Behavior
Should successfully connect and disconnect.
Actual Behavior
Crashes after period of time..
Steps to reproduce
// If possible, attach a picture of your setup/wiring here.
Code to reproduce this issue
// If your code is longer than 30 lines, GIST is preferred.
Debug Logs
Other items if possible
build
folder (note this may contain all the code details and symbols of your project.)