bug: deep sleep not entered if sotacat disconnected from radio with trs still plugged in

[jeffkowalski]

It could be a common occurrence in the field:

1. Use the SOTAcat in conjunction with the radio for a while.
2. QRT, pack up, unplug SOTAcat from radio, but forget to unplug TRS connector also from SOTAcat.
3. SOTAcat remains active, not noticing it's no longer connected to radio
4. Battery drains

Need to ping the radio over the uart from time to time to see it it's still connected. Otherwise, go into deep sleep.

[brianmathews]

Even with the TRS plugged in, it will go to sleep if there is no HTTP request in the time threshold. The reason you might not see it go to sleep is if you are running the browser web page for SOTACAT which checks the battery level (a REST API call) once per minute, and that resets the ESP32 shutdown watchdog such that it can't go to sleep. If you stop your connection to the device, the watchdog will eventually time out and initiate sleep.

So I think this is a feature request to check and see if we are still connected to the radio. To do that we need better management of when we talk to the radio since there are multiple threads that can be running and we need to queue/mutex access to the radio. Some access to the radio has to have a different priority than other access to the radio such as when doing an FT8 synthesis where timing is critical. Other cases aren't as time critical, but because some KX commands can take a LONG time, we can "blow" an FT8 initiation window if the prep commands are interrupted.

[jeffkowalski]

Likely then, this bug relates to #8

The grand solution I'm thinking of is two producer-consumer queues: one to handle the web side, fielding browser requests, queuing them, and awaiting producer response; another to do the same on the radio end. I have to dwell a little on how to make the queues respond synchronously for those callers who are expecting a return with a value, not just a "put your request in queue".

On second thought, maybe simple mutex would be the way to go.

[brianmathews]

...and the mechanism / queue for access to the radio has to have some concept of absolute priority so that I can "skip the line" or "freeze the queue" when doing an FT8 sequence. For example: a call to ask if the radio is still connected or to get the battery level should be at the back of the line compared to FT8 work.

There are a few places where FT8 timing matters:

1. When we get the command that we need to prepare for FT8. There are several settings we have to change on the radio, and some of those settings take a very long time: in the order of seconds. For example, we may need to change bands and all the relays may need to get to the right ATU state, filter state, etc. The timing of this matters is because:
2. The SOTAMAT app has the "clock" and knows when the FT8 "start" time boundary is. It has to coordinate with the ESP32 as to which FT8 window start time we are going to use. If we look at the clock and we have a start window that is 1 second in the future, but the KX relays take longer than 1 second to get into the "prepared" state, then we have a problem. So currently SOTAMAT and SOTACAT have some agreements that when we are close to an FT8 start window, it might be better to delay until the following start window. Any commands that come in (asking the radio if it is awake) can add 100ms to that timing and throw off the algorithm.
3. Once both SOTAMAT and SOTACAT agree on a time to start, SOTACAT has a special FT8 watchdog timer that covers the case when we lose connection to the mobile phone. If something goes wrong, the SOTACAT watchdog times out and it will restore all the KX settings back to their original non-FT8 state. We don't want that watchdog KX communication or timer to be interfered with by some low priority "are you awake" command.
4. Once SOTACAT starts transmitting FT8, we have a rigid 160ms window to pump UART commands with. This seems like a long time considering the device runs in the many MHz, however the ESP-IDF RTOS has a concept of "Ticks", and there are only so many ticks per second (it is programmable, but often 100Hz). Coding is simpler if we use "tick" based delays for the 160ms steps rather than lower level clocks that "fight" with the RTOS. The RTOS allows us to have a high-priority thread that will have absolute priority over lower priority threads, but we don't want other UART activity happening when in the middle of FT8 transmission as it will take a "tick" to task switch.
5. When done we have to restore the radio with the watchdog code. Restoring some settings requires we put the radio in special modes before we can access certain parameters, and then we switch back. We don't want some other thread interrupting that sequence.

[jeffkowalski]

Back to my original reported bug. You're correct, Brian, it's working correctly. After many minutes (AUTO_SHUTDOWN_TIME_SECONDS currently 30 min), the SOTAcat notices inactivity and shuts down, per the code in idle_status_task.

Closing this bug as "working as designed", and moving your last comment to a discussion so we can ponder threading and preserving FT8 timing.