Closed podhrmic closed 2 years ago
Need to be queried with the hearbeat monitor, should be always online
setupEnv.Json
-> cyberPhysNodes
To query these components, do periodically (on the CAN TCP bus):
If the component responds within the TIMEOUT it is considered healthy.
with systemctl status $SERVICE_NAME
CAN_DISPLAY
HACKER_KIOSK
INFOTAINMENT_THIN_CLIENT
To query these components, query them over ssh and parse the response. If the response contains Active: active (running)
then the component is healthy. If the response contains Active: inactive
then the component is not healthy (error).
Occasionally these will be down during reset (keep track of reset requests).
FREERTOS_1,2,3
INFOTAINMENT_SERVER_1,2,3
To query these components, do periodically (on the CAN UDP bus)
If the component responds within the TIMEOUT it is considered healthy. In addition, FreeRTOS might send a CAN_ID_CMD_COMPONENT_ERROR(SENSOR_THROTTLE|SENSOR_BRAKE)
message (on port 5002), in which case we need to request restarting Teensy with CAN_ID_CMD_RESTART(TEENSY)
on the TCP bus. The messages might be coming relatively quickly, so keep track of the reset requests (no more than one at a time).
Occasionally these will be down during reset (keep track of reset requests).
OTA_UPDATE_SERVER_1,2,3
To query these components, send a HTTP request to each server.
Have a class that represents the abstract component (list of components above). Components have a notion of health (states are: HEALTHY/ERROR/maybe WARNING). Not every component is queried the same way, some components need to be tested in more than one way to determine they are healthy.
The monitor collects this information about the system, and prints it on demand.
Fixed in #1263
add simple handling of failures (retry? something else?)(part of #1062 )