PST would benefit from the ability for software features to be able to store and retrieve bits of information. That information could likely be calibration data for on-board or attached hardware, but could conceivably be configuration information, passwords, and so on, for future features. The Data Storage (DS) capability for PST will provide support for software developers who wish to use it for their features.
Vocabulary
For the purposes of following this document, the following definitions/concepts will be used:
Block: Flash storage is in chunks we will call Blocks (e.g. a 4 kbyte Block of Flash)
DS: The overall capability described in this document will be called DS (Data Storage)
Item: An Item is a single value or variable that needs DS capability. For instance, an offset integer used by an ADC feature would be an Item.
Group: A Group will be expected to be smaller than a Block; it will always be of a size in multiples of 16. For the MVP, there will be a single Group per feature that needs DS capability. As an example, if there is a feature called HIRES that needs 15 bytes of DS capability, then the HIRES DS Group will be of a size of (perhaps) 32 bytes (larger than 16, so that there is room for the HIRES feature to comfortably expand its DS needs in future), and that Group will contain a C Structure (described next).
Group data Structure: The Group data structure will be accessible in RAM, and will (when things are consistent) be a copy of the
Group that is within the Block in Flash. The structure will contain things such as a version number and the actual data content (in types such as integer, float and C string), which are the Items. The C structure will always be of a smaller size than the Group size; the C structure is effectively the data-aware interpretation of the group.
High-Level Requirements
As a software developer, access to an API for storing and retrieving data should be available, so that it is possible to maintain persistent, non-volatile information in an easy-to-use way.
A software developer would like to be able to, at a minimum, store and retrieve types of data such as calibration integers, or floating-point values, or strings, so that the end feature can easily incorporate the data items.
A software developer must be able to develop features that can call an API to determine if data is valid or not, so that unexpected values are not used by their feature.
A software developer must be able to store default data if the current stored data is invalid, so that a sane configuration can be made available for new device or new feature and for corrupted memory circumstances.
High-Level Design
Ability for the developer to allocate storage for a feature during compile-time
There should be room available for features to grow in future releases, without trampling on storage allocated for other features.
One approach could be to statically define indexes for the storage, in an array, with one entry per feature. For instance, if two features called HIRES and PWM required storage, then two DS groups would be defined:
#define DS_GROUP_HIRES 0
#define DS_GROUP_PWM 1
#define DS_GROUP_UNALLOCATED 2
// allocate a minimum of 32 bytes, and ensure when adding a new entry, that at least 16 bytes are spare for subsequent enhancements for the feature! And all index values must be on 16 byte boundaries. Example: A value of 18 is _not_valid_!
uint16_t ds_mem_index[] = {0 /* HIRES */,
32 /* PWM */,
64 /* UNALLOCATED */};
Anyone creating a new feature would move the unallocated value.
Note: The linker file may need to be adapted, to avoid that the area is ever overwritten by growing firmware size. GCC allows for it. Optional: it also allows to use the address defined in the linker file in C code._Check the Pico SDK to see what is suggested. Today the Pico SDK examples deliberately choose a block that is right at the end of Flash, so that the code would need to fill all remainder Flash space before that block could be accidentally overwritten.
A separate user story is created for it.
There would be a base address for internal flash, for instance:
#define DS_MEM_FLASH_START 0xffff1000
(edit jc: @shabaz123, I added this area))
issue 34 created a symbol for this area in the linker / loader script. In C code, we can get at the start address via:
in our DS api header, if we write this code (takes no memory space or clock ticks. It is as efficient as using a define)
// only if no pico headers used - normally not required because we'll include the flash headers #include <stdint.h>
inline uint32_t *ds_get_address_persistent() {
extern uint32_t ADDR_PERSISTENT[];
return ADDR_PERSISTENT;
}
When the base address is needed, we can use this (example code)
sprintf(addr, "address = %x", ds_get_address_persistent());
Ability for the developer to define the data
This could be in the form of a C structure, for instance:
Access to the data should ideally be through function calls, instead of directly to the RAM copy. If direct access to the RAM copy was allowed, then in the future it would be hard to make improvements to the mechanism.
These functions would manipulate the RAM copy (i.e. it wouldn’t immediately write to Flash). The ds_store_item function would need to set a flag to indicate that the RAM copy is different to Flash, so that persistent storage is only written to when a change occurs.
The flag could be called (say) ds_nv_write_pending.
The ds_store_item function should check if the new value is the same as existing data, to prevent unnecessary flash writes to a location. The number of writes is limited. The ds_nv_write_pending flag should only be set if any data is different to what is in Flash.
Ability for the RAM copy to be initialized from persistent storage
The initXYZUtils() function for features that use the DS capability should be responsible for calling a function, for instance as follows:
ds_mem_init(DS_GROUP_HIRES);
That function would use (in the MVP) perhaps memcpy, to copy the chunk of Flash into the RAM copy as a block, so that then the ds_get_item function can be used at any time. In future, an enhancement to the ds_mem_init function could be to read from I2C memory if it is present.
The ds_mem_init function should also be responsible for doing a sanity check (at a minimum ensuring a magic number (which can be global, not feature-specific) is present and the version number is correct), and if not, then default values should be written to RAM, and the ds_nv_write_pending flag should be set.
Ability for the RAM copy of all feature storage to be written to persistent storage
Even though each feature may only use (say) 32 or 48 or 64 bytes and so on, of storage, it is not possible to write such a small group of data to Flash; there will be a Flash block size, for instance 4 kbytes. So, all the features data storage (groups) can be written in one go.
At the end of the initInstrument() function (in scpi-def.c), it should be possible to have a function call, such as ds_write_persistent_all(), which will write all the groups into Flash. The function will only do this if the ds_nv_write_pending flag is set, and then the flag would be cleared.
Ability for the developer to write to feature storage at any time
Whenever a feature requires data to become persistent, a function call should be executed, called (say):
ds_write_persistent_group(DS_GROUP_HIRES);
That function could, for the MVP, simply call the ds_write_persistent_all() function.
(jc edit)
possible SCPI commands (suggestion)
// infra
{.pattern = "CALibration:STArt", .callback = calStart,},
{.pattern = "CALibration:END", .callback = calEnd,},
{.pattern = "CALibration:ERAse", .callback = calErase,},
// funtional (examples to show possible signature)
{.pattern = "CALibration:ADC#:VOLTage?", .callback = calAdcVoltQ,},
{.pattern = "CALibration:TEMPERATUREMAXResistance", .callback = calTemperatureMaxResistance,},
Introduction
PST would benefit from the ability for software features to be able to store and retrieve bits of information. That information could likely be calibration data for on-board or attached hardware, but could conceivably be configuration information, passwords, and so on, for future features. The Data Storage (DS) capability for PST will provide support for software developers who wish to use it for their features.
Vocabulary
For the purposes of following this document, the following definitions/concepts will be used:
Block: Flash storage is in chunks we will call Blocks (e.g. a 4 kbyte Block of Flash)
DS: The overall capability described in this document will be called DS (Data Storage)
Item: An Item is a single value or variable that needs DS capability. For instance, an offset integer used by an ADC feature would be an Item.
Group: A Group will be expected to be smaller than a Block; it will always be of a size in multiples of 16. For the MVP, there will be a single Group per feature that needs DS capability. As an example, if there is a feature called HIRES that needs 15 bytes of DS capability, then the HIRES DS Group will be of a size of (perhaps) 32 bytes (larger than 16, so that there is room for the HIRES feature to comfortably expand its DS needs in future), and that Group will contain a C Structure (described next).
Group data Structure: The Group data structure will be accessible in RAM, and will (when things are consistent) be a copy of the Group that is within the Block in Flash. The structure will contain things such as a version number and the actual data content (in types such as integer, float and C string), which are the Items. The C structure will always be of a smaller size than the Group size; the C structure is effectively the data-aware interpretation of the group.
High-Level Requirements
As a software developer, access to an API for storing and retrieving data should be available, so that it is possible to maintain persistent, non-volatile information in an easy-to-use way.
A software developer would like to be able to, at a minimum, store and retrieve types of data such as calibration integers, or floating-point values, or strings, so that the end feature can easily incorporate the data items.
A software developer must be able to develop features that can call an API to determine if data is valid or not, so that unexpected values are not used by their feature.
A software developer must be able to store default data if the current stored data is invalid, so that a sane configuration can be made available for new device or new feature and for corrupted memory circumstances.
High-Level Design
Ability for the developer to allocate storage for a feature during compile-time
There should be room available for features to grow in future releases, without trampling on storage allocated for other features. One approach could be to statically define indexes for the storage, in an array, with one entry per feature. For instance, if two features called HIRES and PWM required storage, then two DS groups would be defined:
Anyone creating a new feature would move the unallocated value.
Note: The linker file may need to be adapted, to avoid that the area is ever overwritten by growing firmware size. GCC allows for it. Optional: it also allows to use the address defined in the linker file in C code._Check the Pico SDK to see what is suggested. Today the Pico SDK examples deliberately choose a block that is right at the end of Flash, so that the code would need to fill all remainder Flash space before that block could be accidentally overwritten. A separate user story is created for it.
There would be a base address for internal flash, for instance:
(edit jc: @shabaz123, I added this area)) issue 34 created a symbol for this area in the linker / loader script. In C code, we can get at the start address via:
in our DS api header, if we write this code (takes no memory space or clock ticks. It is as efficient as using a define)
When the base address is needed, we can use this (example code)
sprintf(addr, "address = %x", ds_get_address_persistent());Ability for the developer to define the data
This could be in the form of a C structure, for instance:
A version value should be mandatory, perhaps integer type.
For a MVP, all persistent data could be copied in RAM, i.e. global variables:
To make it easier to access the data, definitions could be written, for instance:
Ability for the developer to get and store data
Access to the data should ideally be through function calls, instead of directly to the RAM copy. If direct access to the RAM copy was allowed, then in the future it would be hard to make improvements to the mechanism.
Example function calls could be:
Storing an item:
Retrieving an item:
These functions would manipulate the RAM copy (i.e. it wouldn’t immediately write to Flash). The ds_store_item function would need to set a flag to indicate that the RAM copy is different to Flash, so that persistent storage is only written to when a change occurs.
The flag could be called (say) ds_nv_write_pending.
The ds_store_item function should check if the new value is the same as existing data, to prevent unnecessary flash writes to a location. The number of writes is limited. The ds_nv_write_pending flag should only be set if any data is different to what is in Flash.
Ability for the RAM copy to be initialized from persistent storage
The initXYZUtils() function for features that use the DS capability should be responsible for calling a function, for instance as follows:
That function would use (in the MVP) perhaps memcpy, to copy the chunk of Flash into the RAM copy as a block, so that then the ds_get_item function can be used at any time. In future, an enhancement to the ds_mem_init function could be to read from I2C memory if it is present.
The ds_mem_init function should also be responsible for doing a sanity check (at a minimum ensuring a magic number (which can be global, not feature-specific) is present and the version number is correct), and if not, then default values should be written to RAM, and the ds_nv_write_pending flag should be set.
Ability for the RAM copy of all feature storage to be written to persistent storage
Even though each feature may only use (say) 32 or 48 or 64 bytes and so on, of storage, it is not possible to write such a small group of data to Flash; there will be a Flash block size, for instance 4 kbytes. So, all the features data storage (groups) can be written in one go.
At the end of the initInstrument() function (in scpi-def.c), it should be possible to have a function call, such as ds_write_persistent_all(), which will write all the groups into Flash. The function will only do this if the ds_nv_write_pending flag is set, and then the flag would be cleared.
Ability for the developer to write to feature storage at any time
Whenever a feature requires data to become persistent, a function call should be executed, called (say):
That function could, for the MVP, simply call the ds_write_persistent_all() function.
(jc edit)
possible SCPI commands (suggestion)