Implement alarms and event logging in the firmware

[ethanjli]

This mega-PR contains a lot of changes everywhere (I'll try to split things up into smaller PRs in the future!). The most significant one is that the alarms and event logging and list synchronization code from the backend have now also been ported to the firmware, so we have alarms working in fully-integrated system with real hardware.

Firmware:

• I've updated Pufferfish::Application::States to include message types for the event log
• I've ported the backend's ventserver.protocols.application.lists.SendSynchronizer to Pufferfish::Application::Protocols::Lists::ListSender. This implementation differs from the backend's implementation in that this implementation discards each element whose receipt has been acknowledged by the backend (via the ExpectedLogEvent message), because the STM32 doesn't have enough RAM to keep all events in memory.
• I've ported the backend's ventserver.simulation.logs to Pufferfish::Application::LogEvents::IncrementalEventsSender. This is a small wrapper around the list sender which ensures that event IDs increment for each event, and also that event timestamps are correct.
• I've ported the backend's ventserver.simulation.alarm_limits to Pufferfish::Driver::BreathingCircuit::AlarmLimitsService. This handles AlarmLimitsRequests and updates AlarmLimits accordingly.
• I've ported the backend's ventserver.simulation.alarms to Pufferfish::Driver::BreathingCircuit::AlarmsService. This checks sensor measurements against alarm limits and activates (and deactivates) alarm events accordingly. We will need to modify the implementations of these things in the future to have a timeout for how long a value is out of range before the alarm activates or deactivates - essentially, "debouncing" the alarms. This will be needed because when we adjust the FiO2 and flow settings, the firmware will automatically adjust the alarm limits; and initially they will be out-of-range while FiO2 and flow rate adjust to the new settings values, but during this initial period we don't want to trigger an alarm.
• I've refactored Pufferfish::Driver::BreathingCircuit::ParametersService. Everything should still work correctly, but I might have introduced regressions. It would be great if people could test various input values parameters to see if anything breaks, e.g. if invalid inputs are allowed or valid inputs are prohibited/ignored by the firmware.
• I've updated the firmware to send heart rate measurements from the Nonin sensor, so now the frontend displays actual heart rate measurements
• I've generalized Util::RingBuffer to work not just on uint8_ts, but also on other types. I've also added new functionality for RingBuffers not used with interrupts (i.e. every use besides in UARTBuffers), namely look-ahead peek and size reporting. These have not been thoroughly tested, so additional testing and bug-hunting of creating events (the event log uses those RingBuffer functionalities) in different temporal patterns would be appreciated.
• I've added an erase method to Pufferfish::Util::Vector. The erase method is used by Util::OrderedMap, described next.
• I've added a Pufferfish::Util::OrderedMap as a basic associative key-value container (backed by a Pufferfish::Util::Vector). I'd appreciate review of the function interface and feedback on the safety and usability of the method interface provided by this class. I also haven't tested this thoroughly, so additional testing and bug-hunting of activating and deactivating alarm conditions (which updates an OrderedMap) in various temporal patterns would be appreciated.

Backend:

• ventserver.application no longer initializes all values of all state segments to 0 on program startup. Instead, it initializes them to None so that they can be set by the frontend and/or firmware (and in the future initial defaults will be set but they will all be overwritten by the filesystem)
• ventserver.protocols.application.lists.SendSynchronizer now accepts resetting the ExpectedLogEvent.id field to 0. This is because if the frontend restarts after a crash, it will reset ExpectedLogEvent.id to 0 and it should load all logged events.
• ventserver.protocols.backend.ReceiveFilter now has some member objects which handle for receiving log events from the MCU (which is a prerequisite for #310) and sending new events to the frontend; the MCU's event IDs and relative timestamps are first translated into the backend's event IDs and Unix epoch timestamps (which fixes #311, but relies on time remapping only being done on timestamps in LogEvents, not on timestamps in any other messages), and some bugs are fixed related to flooding of event logs.
• I've refactored the implementations of the alarm limits and parameters services in the backend simulator. Everything should still work correctly, but I might have introduced regressions. It would be great if people could test various input values for alarm limits and parameters to see if anything breaks; however, since this is only used for the backend simulator, rigorous testing is not as urgent here.

Frontend:

• I've modified the reducers for ExpectedLogEvent and NextLogEvents so that receipt of new log events will immediately trigger an increase to ExpectedLogEvent.id, rather than the Logs Page triggering this increase through a dispatch. This fixes a bug where if I didn't open the Event Log or Active Alarms modal at least once in the frontend, then new alarms wouldn't trigger the topbar and alarm border until I did open that modal. However, it does expose some issue related to the frontend switching from the dashboard back to the quick start screen and then back to the dashboard when the backend is restarted, under certain conditions. I think this is a separate issue related to state initialization; it may or may not be resolved together with #312.
• I've modified LogsPage so that it doesn't try to sort elements by timestamp, but rather just keeps the ID-based order of log events. This is because changing the clock will happen frequently and can completely mess up a timestamp-based sort order, while ID order is guaranteed to be chronological until uint32 rollover (i.e. after ~4 billion log events; if events are generated at 1/second, it would take 126 years to reach rollover). So it's better to just use ID order.

Protobuf schemas:

• I've changed timestamps from uint32 to uint64, so that we can store Unix epoch timestamps as integers with millisecond precision

I would appreciate testing of the software on this PR - please report all problems you see, and please test it in various configurations (with different combinations of computers, hardware, ventserver.application, and ventserver.simulator); we will fix smaller problems related to changes from this PR in this PR before merging, and we will create Github issues for larger problems or unrelated problems to address in future PRs. We will merge this in as prototype-quality code, but we should identify any regressions or breakage caused here and either fix it here or prioritize it for the next PR.

[rohanpurohit]

Running the tests on my local machine:

backend running ventserver.application:

• start ventilation from the quickstart page did not start the ventilation even after multiple triggers.
• press start ventilation -> refresh page -> goes to the dashboard -> press pause -> goes back to quickstart but button still says pause ventilation
• events log modal was empty (backend was running) no logs for usual things like 'ventilation started' etc
• changing the alarm limits did not register in the events log modal
• stopping the backend gives 'software connection lost' along with the alarm, but no entry in the events log modal or active alarms modal

(not sure if I was supposed to change anything for application)

backend running ventserver.simulator:

• on triggering spo2_too_high the alarm was registered in active_alarms modal and event logs too, there was no sound like when the backend is stopped and there is a sound for software connection lost
• fio2_too_lowalarm was active when alarm limits were changed to trigger an alarm for spo2_too_low which flooded the events log modal with 20-30 entries for one trigger? and similarly, the events log modal kept flooding for subsequent triggers.

Running the tests on the ventilator unit:

backend running ventserver.application:

• events log modal shows the initial entries like 'fio2_limits_changed', ventilation stopped etc correctly.
• unit was without a spo2 sensor therefore it was simulating, changing spo2_alarm_limits triggered alarms for both spo2_too_low and spo2_too_high
• only one entry created per change in the events log modal, active_alarms also displayed the active alarms correctly.
• for example, the alarm for spo2_too_low was triggered when the lower_limit was 54 and the value on the dashboard was 54, should it be triggered when it's 53?
• as there was no air/o2/flow there was no alarm for fio2_too_low? is this expected behaviour? on changing the fio2 lower limit to greater than 0 also did not trigger an alarm for fio2, cannot test with actual flow.
• Alarms for HR were triggered properly for both lower and upper limits as expected. same as spo2 when equal to the lower and upper limits, expected behaviour?

backend running ventserver.simulator:

• No alarms were triggered for HR
• issues like events log modal flooding persisted
• on starting the front end, there is no red border when the alarm sound is on mute, toggling multiple times fixes the issue.

[ethanjli]

This is extremely helpful, thanks for these details!

Backend running ventserver.application on your local machine: I think I'll need to understand your test conditions more thoroughly in order to understand what's happening (for example, was a Nucleo board connected up together with all the sensors). Let's discuss this in today's call. If you're able to screenshare what you see when you run this test during the meeting, that'll be very helpful.

Backend running ventserver.simulator on your local machine:

• Yes, the frontend only makes a sound if it detects that the backend isn't talking to it. For all other alarms, the sound should be generated by the microcontroller (the firmware functionality for this is not yet implemented).
• For the FiO2 and SpO2 alarms I'll need to understand the test conditions (e.g. what were the FiO2 and SpO2 measurements, and what were the alarm limits you set) - this is another one which would be good if you can screenshare while reproducing the test during our next call.

Backend running ventserver.application on the ventilator:

• Let's reproduce the SpO2 too high and too low simultaneous alarms during our meeting so I can troubleshoot.
• When an alarm limit is (e.g. 54) and the frontend shows the value as the same (e.g. 54) and an alarm triggers, this is probably because the frontend is rounding values for display while the firmware is not. So for example, if the alarm limit is 54, the actual value is probably like 53.8, and the frontend is rounding that to 54 for display. This would be a good one for us to discuss together what the correct behavior should be, and addressed in a future PR, because I don't have an answer yet, and there are the usability implications you pointed out.
• FiO2 and flow rate alarms are a special case, as the firmware should automatically set alarm limits based on the settings (e.g. if FiO2 is requested to be 54%, the firmware might automatically set alarm limits of 53%-55%). For this PR I believe I'm setting FiO2 and flow rate alarm limits such that an alarm should never be triggered.

Backend running ventserver.simulator on the ventilator:

• I'm very interested to see what events show up when the events log modal gets flooded. I had discovered some sort of bug related to flooding of the events log with "Ventilation started" and "Ventilation stopped" events when the backend was force-restarted, and your observation may or may not be related. Let's try to reproduce this in our call.
• When alarms are muted, the border should go away; it should not be shown until alarms are unmuted, so it sounds like you might've discovered some buggy behavior when toggling the button multiple times. Let's try to reproduce this in our call.

[ethanjli]

Things to do in future PRs or issues:

• Create an issue for the backend's ventserver.application to raise an alarm when the MCU cannot be detected.
• Create an issue for driving audio sounds for alarms in the firmware
• There is weird behavior when the backend is restarted but the frontend is not, e.g. with the events log.
• Add debouncing behavior for the alarms. We also need signal smoothing of values for alarms, as well as rounding for alarms, so that the frontend's displayed value is consistent with the alarm activation/inactivation behavior, in edge cases where roundoff makes a difference.
• @Sudhir-dev We need a confirmation dialog on the "Apply Changes" button in the Set Alarms screen.
• Automatically set FiO2 and flow rate alarm limits based on FiO2 and flow rate settings.
• @Sudhir-dev Should it be possible to open the quickstart screen from the launch screen even if the backend hasn't started? Relatedly, should we give some user feedback if the backend is not up yet? (e.g. when we press the Start button, it changes to "Loading" or brings up a "Loading" modal until the backend is confirmed to be available, after which the app goes to the quick start screen)

Issues to try to reproduce and document so that Ethan can reproduce:

• "stopping the backend gives 'software connection lost' along with the alarm, but no entry in the events log modal or active alarms modal"

Small fixes for this PR:

• Add HR alarms into the backend simulator alarms
• Add HR simulation into the firmware simulator when HR is not available, or at least do the same thing as SpO2-related simulation for HR.

[ethanjli]

@rohanpurohit I've solved some issues with flooding of the event log. When you have a chance, can you play around with b128343 to see if the issues you previously saw with flooding the events log have been fixed, and/or if there are other issues with the events log, e.g. when the backend is restarted?

Can you also see whether/how you can reproduce the "stopping the backend gives 'software connection lost' along with the alarm, but no entry in the events log modal or active alarms modal" behavior you had observed? I've changed some things in the frontend which might interact with the code related to this behavior, but nothing which I'd expect to have fixed this issue.

[rohanpurohit]

Running the tests on my local machine:

backend running ventserver.application:

• press start ventilation -> refresh page -> goes to the dashboard -> press pause -> goes back to quickstart but button still says pause ventilation

(Test env- local machine without an STM32 connected to it)

• This behavior is fixed now

Can you also see whether/how you can reproduce the "stopping the backend gives 'software connection lost' along with the alarm, but no entry in the events log modal or active alarms modal" behavior you had observed? I've changed some things in the frontend which might interact with the code related to this behavior, but nothing which I'd expect to have fixed this issue.

I have not been able to reproduce this behavior since, maybe we can rule it out for now? I'm not sure under what circumstances I saw that behavior

@rohanpurohit I've solved some issues with flooding of the event log. When you have a chance, can you play around with b128343 to see if the issues you previously saw with flooding the events log have been fixed, and/or if there are other issues with the events log, e.g. when the backend is restarted?

The flooding issue seems to have improved, but I'd say it's still there?. unfortunately, I don't have access to a ventilator unit or STM32 connected to sensors therefore can't test it in application mode, although I had seen similar flooding in the previous test on application too.

To reproduce the behavior on backend running ventserver.simulator:

• start ventilation -> change spo2 alarm_limits from 21/100 to 56/100 -> wait for the simulation to reach around 50-54 -> change the lower limit of spo2 to 54
• Observation : the number of event log entries created for border-line values has reduced,
• for example : running the test for 5 mins with the same alarm_limits would have created ~200 to 300 entries whereas now it's ~100 to 120

[ethanjli]

Ok, that looks good. There should be flooding on alarms when measurements are very near alarm limits, because I haven't fixed that (progress on that will be tracked in #322). If I fully fixed the other flooding issues, there shouldn't be flooding on other events (e.g. parameter changes or alarm limits changes).

[ethanjli]

@rohanpurohit In this PR I've defined more MessageTypes, and once I merged develop in, some of your MessageTypes parse tests, which assume that no message types have values above 7, have started failing. We'll need those tests to work even after I define more message types, so maybe the unit tests should have their own definition of the MessageTypes enum. How do you want to proceed?

[ethanjli]

Because this PR has been reasonably well-tested (though testing on the final refactoring and fixes has been much lighter) and is part of the v0.7 milestone, I'm going to merge this into develop as-is, and as prototype-grade code. For records-keeping:

1. This project is licensed under Apache License v2.0 for any software, and Solderpad Hardware License v2.1 for any hardware - do you agree that your contributions to this project will be under these licenses, too? Yes
2. Were any of these contributions also part of work you did for an employer or a client? No
3. Does this work include, or is it based on, any third-party work which you did not create? No