Open AlexXZero opened 2 years ago
AlexXZero
commented
2 years ago From my experience usually eventlog allow storing a few weeks history of events, but it depends on how often we write the events. It also possible to use a few eventlogs (like system log and user log) which will have different time of storing history, e.g. a few months for system log and a week for user log.
JF002
commented
2 years ago @AlexXZero Thanks for this great analysis! We do plan on storing "historical" data for the step counter and heart rate sensors, for example. I created this project for that purpose. Your suggestion here is much more generic and introduces new data (event, crash dump, battery,...) which might be helpful for debug, improvements, and for users in general! Just keep in mind we will have to find a way to expose those data to the users (via the UI or companion app).
So yeah, that's nice, especially because I don't think anyone has started working on it ;-)
Regarding the low-level implementation, using the existing filesystem seems to be the most practical solution, as LittleFS already implements some kind of wear leveling and provides higher level APIs.
Verification
Pitch us your idea!
Storing history of events in flash memory
Description
Background
Hi there, I'm embedded software engineer and I like to use PineTime watch. It is my first created feature for open source project, so let me know if I do something wrong with this ticket. I would like to help with adding lots of different features like showing step counter history (#788), showing battery percentage history (on watch), storing ble debug events, etc.
But I found that most of them require storing different events to non volatile memory. I have experience with using event log for these purpose. I know that this feature is not visible for end-user, but it is helpful for implementing lots other features.
Feature description
It is possible to store lots of events on flash memory using ring buffer (i.e. old events will be overwritten by new events).
Usage:
1. History of battery voltage
We can store battery voltage value each an hour, then we can build a plot for showing battery discharging during some period of time.
2. History of step counter
It is the same idea as with battery voltage, but we should store step counter. We can store it once a day or more often.
3. Crash dump
We can use event log for storing events with reason of fault in case if hard fault was detected.
4. Debugging
I found it useful to store some events like BLE connection, low battery level, unexpected state of some subsystem, etc to eventlog. Then this data can be used for signalizing some errors or unexpected behavior.
4.5 Storing startup data
For debugging purpose we can store some startup up data, like FW version, battery voltage level, reset reason, etc. then this data can be used for signalizing some issues as it provides all history of FW updating, reason of rebooting and other.
5. Capture user data for getting some statistics
It is possible to add some test mode which allows to save some user data, like accelerometer, HRM, etc. Then these data might be used for improving step counting algorithm or some other subsystem.
Implementation idea (generic part)
4byte event alignment
We have a few
Xkb pages for storing Eventlog (at least 2). Each event takes at least 4 bytes for storing it. First bit of each 4 bytes is used as flag if it is last part of this event or not (like UTF-8). (Most events should use only 4 bytes, later we can allocate more ranges for storing long events). The rest bits of first byte is used for selecting event type. And the rest bytes are used for event data. Examples:80 00 00 05// Event type = 0, value = 50ms. Timestamp. This type of events uses the rest 3 bytes for storing time (10 ms units) which elapsed from last event. So we should save this event before storing each another new event in case if elapsed more than 10ms.81 00 00 00// Event type = 1, value = 0. Single event. This type of event uses the rest 3 bytes (24kk combinations) for storing a single events like BLE client connected, BLE client disconected or Battery voltage was dropped less than 2 volt. Usually I prefer to split 24kk values to some ranges (normal events, warning events, debug events)02 00 00 00 80 00 00 00// Event type = 2, value = 0 (6 bytes + 7 bit) this is example of long event. You can see that it uses 1 bit of each 4 bytes as sing of end of the event. (like UTF-8).82 00 00 02// Event type = 2, value = 2. Eventlog page swapping event. This event is used after swapping page of eventlog. data of this event shows how many page swapping occurs for all time. It also used for detecting the last event page on initialization.ff ff ff ff// Event type 7f, value ff ff ff. This is a fake event which means that the rest area of eventlog page is free.Ring buffer usage
Ring buffer is used for erasing older events and storing newer events. Example: we use 6 pages 4 kB each of them. It means that we are able to store 1024 events per each page (Actually I reserve 24 events so we have 1000 events for storing per page). On initialization we read first event of each event page to see event with page swapping which have highest value for latest eventlog page. Then we search event
0xffffffffwhich means that the rest events of this page are free. In case if we add 1000th event we start eventlog procedure which should erase oldest eventlog page, then write eventlog page swapping event to this page. Now we are able to use this page for storing new events. All events which occurs during page swapping should be stored to the reserved area of last eventlog page (24 events).Reserve a few events at the end.
Usually I reserve some space at the end of each page for storing crash dump which might occurring during page swapping when new page is not available for storing eventlog.
Option for implementation:
1. Using a few pages of internal nrf52 micro flash memory.
I have experience of working with this uC and usually I use a few pages of internal flash memory for storing eventlog. I know that you are using MCU bootloader which probably erase all this memory during FW updating. Probably it is fine, but it means that we can't receive history of FW updating versions, etc.
Cons:
2. Using some low level API for storing events in SPI memory.
Cons:
3. Use existing FS API for storing eventlog to the file
Cons: