SIMCOM: Reset modem when framing is lost

[markwj]

In the event we receive framing errors, the mux channel is invalid and will probably not work any more. In such cases, perhaps the best is to reset the modem. Or have an overall timeout to reset the modem if we don't receive good mux traffic within a period.

[dexterbg]

Annoyingly, UART and MUX framing errors are still common & frequent on my module, despite my work on interrupt optimization last October. The issue is caused by the missing hardware flow control to/from the modem, so isn't really solvable. But the errors normally recover without action on my module:

I (389717506) webserver: HTTP GET /shell
W (389718776) gsm-mux: Frame overflow (2048 bytes)
I (389721096) ovms-server-v2: Send MP-0 S88.4,…
I (389721106) ovms-server-v2: Send MP-0 D0,…
I (389721116) ovms-server-v2: Send MP-0 L51…
I (389721116) ovms-server-v2: Send MP-0 F3.2.010…
I (389722106) ovms-server-v2: Send MP-0 h20082,0,RT-GPS-Log,…
I (389722156) ovms-server-v2: Incoming Msg: MP-0 h20082
W (389766806) simcom: UART hw fifo overflow
W (389767796) simcom: UART hw fifo overflow
W (389779776) gsm-mux: Frame overflow (2048 bytes)
I (389782106) ovms-server-v2: Send MP-0 S88.4,…
…
W (389826806) gsm-mux: Frame error: EOF mismatch (CHAN=12, ADDR=31, CTRL=35, FCS=39, LEN=33)
W (389826816) simcom: UART hw fifo overflow
W (389827186) gsm-mux: Frame error: EOF mismatch (CHAN=1, ADDR=05, CTRL=ff, FCS=3a, LEN=77)
…
W (389947786) gsm-mux: Frame error: EOF mismatch (CHAN=13, ADDR=35, CTRL=31, FCS=45, LEN=30)
W (389947786) simcom: UART hw fifo overflow
W (389948766) gsm-mux: Frame error: EOF mismatch (CHAN=1, ADDR=05, CTRL=ff, FCS=30, LEN=70)
…
W (390007786) simcom: UART hw fifo overflow
W (390020766) gsm-mux: Frame overflow (2048 bytes)
…

[dexterbg]

The occasional MUX freeze issue is also / still present with the SIMCOM 7600, although frame error frequency normally is low.

There's another issue arising from these: the network layer apparently assumes the connection is still valid, transmissions appear to succeed normally. Effectively all transmissions without an ACK handshake (i.e. all standard MP messages except history data, probably also all MQTT updates(?)) are lost during these events.

In this recent case, the trip report data history entry was retransmitted after modem restart, but the text notification was lost:

2022-06-03 16:23:18.379 CEST W (1027390399) gsm-mux: Frame error: EOF mismatch (CHAN=8, ADDR=20, CTRL=45, FCS=31, LEN=41)
2022-06-03 16:23:18.479 CEST W (1027390499) cellular: UART hw fifo overflow
…
2022-06-03 16:37:40.139 CEST I (1028252159) cellular: Signal Quality is: 6 (-101 dBm)
2022-06-03 16:37:46.139 CEST I (1028258159) cellular: Signal Quality is: 14 (-85 dBm)
2022-06-03 16:37:52.139 CEST I (1028264159) cellular: Signal Quality is: 9 (-95 dBm)
2022-06-03 16:37:55.139 CEST I (1028267159) cellular: Signal Quality is: 14 (-85 dBm)
2022-06-03 16:38:34.159 CEST I (1028306179) cellular: Signal Quality is: 10 (-93 dBm)
…
2022-06-03 16:39:41.379 CEST D (1028373399) events: Signal(vehicle.off)
2022-06-03 16:39:44.349 CEST D (1028376369) events: Signal(notify.info.drive.trip.report)
…
2022-06-03 16:41:38.319 CEST W (1028490339) cellular: 3 minutes since last MUX rx frame - assume MUX has failed
2022-06-03 16:41:38.319 CEST I (1028490339) gsm-ppp: Shutting down (hard)...
2022-06-03 16:41:38.319 CEST I (1028490339) gsm-ppp: StatusCallBack: User Interrupt
2022-06-03 16:41:38.319 CEST I (1028490339) gsm-ppp: PPP is shutdown
2022-06-03 16:41:38.319 CEST I (1028490339) gsm-nmea: Shutdown (direct)
2022-06-03 16:41:38.319 CEST I (1028490339) SIM7600: Power Cycle (SIM7600) 2000ms

MP location updates received on server:
2022-06-03 16:38:07
2022-06-03 16:38:18
2022-06-03 16:38:29
…
2022-06-03 16:42:32

There were no signs of the MUX failure before the timeout.