Fruity Mesh: How to save data in Flash

ravindern13 commented 2 years ago

I want to save some data in the flash. Could you please suggest me how to save the data in the Flash?

siretty commented 2 years ago

Hello @ravindern13,

FruityMesh includes the RecordStorage type which is instanciated as a member of GlobalState.

As an example, the EnrollmentModule uses the record storage for the enrollment information.

Thank you for your interest!

Kind regards Daniel

Edit: Slight change in wording.

siretty commented 2 years ago

An overview of the RecordStorage can be found in the accompanying documentation.

PriyeshNema commented 2 years ago

Hi @siretty

I'm also working with @ravindern13 on this. To summarise our requirement, we are having a custom module (a vendor module) and we need to store some custom data to flash, which we want to read from flash during boot-up. I did make a configuration structure as suggested in the custom module creation tutorial, and added the parameters we require (keeping in mind the 4 byte alingment).

I have also declared a recordId using this->recordId = 501; (in my module Init, is it correct?). I call the SaveRecord() when I want to save the data to flash and pass this recordId. I also assume that this gets read as part of bootup and I may need to catch my data in MyModule::COnfigurationLoadedHandler(), am I correct with this assumption? But this handler is never called for my module. What should be done to get this handler called in my custom module

snipping of the log when I write to flash 00> [ORMSModule.cpp@192 ORMSMOD]: Setting Pairing Status to 1 00> [RecordStorage.cpp@183 RS]: SaveRecord id 501, len 19 00> [FlashStorage.cpp@124 FLASH]: Queue CachedWrite 537129408 to 1044484 (28) 00> [FlashStorage.cpp@197 FLASH]: processing command 2 00> [FlashStorage.cpp@248 FLASH]: copy cached data to 1044484, length 28 00> [FlashStorage.cpp@261 ERROR]: Flash operation returned 17 00> Connection secured: role: 559672, conn_handle: 0x1, procedure: 0. 00> [FlashStorage.cpp@303 FLASH]: Flash operation success

Also, I always get 00> [Config.cpp@272 CONFIG]: No flash config for module 7 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 3 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 1 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 2 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 5 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 6 found, using defaults

messages whenever the device is booted. Why there is no config for even other modules in flash?

siretty commented 2 years ago

Hello @PriyeshNema,

if you can use the module configuration, you can make your life easier by using the following utility function to save the module configuration in the RecordStorage:

Utility::SaveModuleSettingsToFlash(this, this->configurationPointer, this->configurationLength, nullptr, 0, nullptr, 0);

It wraps the SaveRecord method for use with module configuration.

Please also make sure that in your featureset the module is added and explicitly sets the (correct) record storage id:

size += GS->InitializeModule<YourVendorModule>(createModule, RECORD_STORAGE_RECORD_ID_VENDOR_MODULE_CONFIG_BASE + 1);

This line sets the recordId for you and you should not set it explicitly in the module initialization.

PriyeshNema commented 2 years ago

Thanks @siretty (Daniel)

I was not aware on how to provide the record ID, neither this was mentioned in any tutorial. It will be a good information to be mentioned in tutorials (suggestion).

Will try this out and will let you know.

PriyeshNema commented 2 years ago

Hi @siretty

I did try as suggested. I'm now getting a call to my module's ConfigurationLoadedHandler(), but unable to get my loaded configuration back. My observations:

1) Still getting following messages: 00> [Config.cpp@272 CONFIG]: No flash config for module 7 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 3 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 1 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 2 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 5 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 6 found, using defaults 00> [Config.cpp@272 CONFIG]: No flash config for module 0xFFFFFFF5 found, using defaults 00> [ORMSModule.cpp@115 ORMSMOD]: ConfigurationLoadedHandler migratableConfigLength 0

2) Do I need to do something special in ConfigurationLoadedHandler() to get my specific configuration back into my structure? I have declared the following structure for my module:

#pragma pack(push)
#pragma pack(1)
//Module configuration that is saved persistently (size must be multiple of 4)
struct ORMSModuleConfiguration : VendorModuleConfiguration 
{
        //Insert more persistent config values here
        u8 exampleValue;
        u8 iAmPaired;
        ble_gap_addr_t onityLockMACAddr;
        u16 onityGAPConnHandle;
};

Instancaited in module class as:

    //Declare the configuration used for this module
    DECLARE_CONFIG_AND_PACKED_STRUCT(ORMSModuleConfiguration);

Edited - for better code styling

PriyeshNema commented 2 years ago

@siretty

If possible, can you please have a look at this on priority. We are in some tight situation.

Thanks

mariusheil commented 2 years ago

I, I hope that I will have some time to check this today, but for now just a quick idea, not sure if it helps. I did not yet have the time to read the above discussion properly.

I think a migratable config is just passed in in case your configuration version has changed and needs to be migrated from a previous version. You will have your config placed in the config struct if it was loaded successfully.

What looks wrong according to your log is the following: 00> [Config.cpp@272 CONFIG]: No flash config for module 0xFFFFFFF5 found, using defaults

Your vendor moduleId looks wrong. Check the documentation on vendor module Ids. Do you actually have a valid company identifier registered with the BLE SIG? If not, check https://www.bluerange.io/docs/fruitymesh/Specification.html#_modules also check the tutorial https://www.bluerange.io/docs/fruitymesh/ImplementingCustomModule.html#_creating_a_ping_module on how to generate a correct vendor module id.

Maybe that helps. I will add a new comment if I have the time to check this more closely.

PriyeshNema commented 2 years ago

Hi @mariusheil

yes, I do understand regarding the version change and the mi9gratable config which is passed. But in our case, we do not change the version, atleast from our end. So, in our configuration structure, we just save the pairing status (true or false) and save the paired MAC of the device. This is done generally once when we pair. But the main thing is to read back the pairing status and the MAC when our device reboots.

Regarding the module ID, i do not see any issue there as I have tried it with the company ID as well. The module ID we have kept for our module is 245 (decimal) in PrimitiveTypes.h. This is being wrapped by FruityMesh with something else and eventually it gets printed like 0xFFFFFFF5. Our company ID is registered and I have tried with that module ID as well, and results are same.

My other question is, leave apart my custom module, this message is also coming for core module. Does the core modules don't save configuration in flash? If not, I don't worry, I just want my module's configuration to be loaded back from flash in some way. Please help here

PriyeshNema commented 2 years ago

Anything further on this? @siretty @mariusheil

mariusheil commented 2 years ago

Hi, sorry, but I did not yet find the time to reproduce your issue and @siretty will only be available from monday. You should however be able to find the issue yourself I would say. Here are a few tips:

Make sure logtags for RecordStorage (RS) and FlashStorage (flash) are enabled, but as you posted a log above, that looks fine already
Use the utility method that @siretty mentioned above to store the record and check the log output again
Use the command "getrec", e.g. "getrec 501" to check if your data has been stored (see here https://www.bluerange.io/docs/fruitymesh/DebugModule.html#_saving_retrieving_records)
Add some log output, e.g. with logtag "ERROR" as it is always displayed to this method: Module::LoadModuleConfigurationAndStart that prints out the moduleId of the current module and also prints out the record storage id so you can see which record it tries to load upon start
Maybe also add some logs to Conf::LoadSettingsFromFlash or debug the method to check what goes wrong.

As for your other questions:

The moduleId you chose was taken from the reserved range, Everything above 240 is reserved for extensions, so that could be the issue. As I have seen from your configuration above, you seem to have followed the tutorial and used the Vendor Module implementation, this requires a proper vendorId, please construct your moduleId e.g. like this: ** constexpr VendorModuleId VENDOR_TEMPLATE_MODULE_ID = GET_VENDOR_MODULE_ID(0xABCD, 1);
In case there is trouble with this vendor module Id, you could use moduleId 200 for testing which is ok for experimenting, but you will need to remove everything from your module that uses "Vendor" e.g. in the configuration and while parsing messages which is what all the core modules do.
The core modules do not save any settings, we try to avoid storing settings in flash whenever possible as this will make the devices much more failsafe.

If you have trouble storing the module config and you are in a hurry, just use the save and load methods in the same way the DebugModule does and use the recordId 501 that you already chose.

Marius

PriyeshNema commented 2 years ago

00> [Module.cpp@66 ERROR]: moduleId : -249 Record ID : 7
00> [Config.cpp@272 CONFIG]: No flash config for module 7 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -253 Record ID : 3
00> [Config.cpp@272 CONFIG]: No flash config for module 3 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -255 Record ID : 1
00> [Config.cpp@272 CONFIG]: No flash config for module 1 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -254 Record ID : 2
00> [Config.cpp@272 CONFIG]: No flash config for module 2 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -251 Record ID : 5
00> [Config.cpp@272 CONFIG]: No flash config for module 5 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -250 Record ID : 6
00> [Config.cpp@272 CONFIG]: No flash config for module 6 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : 32768496 Record ID : 501
00> [Config.cpp@272 CONFIG]: No flash config for module 0x01F401F0 found, using defaults
...
...
00> [ORMSModule.cpp@171 ORMSMOD]: Setting Pairing Status to 1
00> [RecordStorage.cpp@183 RS]: SaveRecord id 501, len 20
00> [FlashStorage.cpp@124 FLASH]: Queue CachedWrite 537129368 to 1044484 (28)
00> [FlashStorage.cpp@197 FLASH]: processing command 2
00> [FlashStorage.cpp@248 FLASH]: copy cached data to 1044484, length 28
00> [FlashStorage.cpp@261 ERROR]: Flash operation returned 17
00> Connection secured: role: 555688, conn_handle: 0x1, procedure: 0. 
00> [FlashStorage.cpp@303 FLASH]: Flash operation success
...
  < getrec 501
00> Record not found

<reboot>

...
00> [Module.cpp@66 ERROR]: moduleId : -249 Record ID : 7
00> [Config.cpp@272 CONFIG]: No flash config for module 7 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -253 Record ID : 3
00> [Config.cpp@272 CONFIG]: No flash config for module 3 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -255 Record ID : 1
00> [Config.cpp@272 CONFIG]: No flash config for module 1 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -254 Record ID : 2
00> [Config.cpp@272 CONFIG]: No flash config for module 2 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -251 Record ID : 5
00> [Config.cpp@272 CONFIG]: No flash config for module 5 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : -250 Record ID : 6
00> [Config.cpp@272 CONFIG]: No flash config for module 6 found, using defaults
00> [Module.cpp@66 ERROR]: moduleId : 32768496 Record ID : 501
00> [Config.cpp@272 CONFIG]: No flash config for module 0x01F401F0 found, using defaults

Do you see any issue with above logs?

mariusheil commented 2 years ago

Hello,

I could indeed see an issue: 00> [FlashStorage.cpp@261 ERROR]: Flash operation returned 17

=> Error 17 means BUSY, which means that the SoftDevice was not able to write to flash correctly. I am not sure why success is returned afterwards, this might be a bug in our FlashStorage class, I will put that in our internal bug tracker.

But the real issue is probably that your device is too busy to write to flash. (Usually FruityMesh will do a few retries before it gives up, but maybe we have a bug in there.) Your device could be too busy handling other connections or doing other stuff. Can you try to write without any active connections? The SoftDevice will not be able to write to flash under certain conditions, but that generally means that the device is under too much load.

Marius

PriyeshNema commented 2 years ago

I also tried to put some logs inside LoadSettingsFromFlash(). Somewhat difficult to print the length due to class implementation.

I implemented this way. Can you tell if it's correct? Do mention the correction

        if(Utility::IsVendorModuleId(module->vendorModuleId))
        {
            logt("ERROR","Data set 1 moduleId : %d Module version : %d length %d", 
                    ((VendorModuleConfiguration*)configurationPointer)->moduleId, 
                    ((VendorModuleConfiguration*)configurationPointer)->moduleVersion,
                    configurationLength
                    );

            logt("ERROR","Data set 2 moduleId : %d Module version : %d length %d", 
                    vendorModuleConfig->moduleId , 
                    vendorModuleConfig->moduleVersion,
                    configData.length.GetRaw()
                    );
        }

I got the log as below

00> [Module.cpp@66 ERROR]: moduleId : 32768496 Record ID : 501
00> [Config.cpp@243 ERROR]: Data set 1 moduleId : 32768496 Module version : 1 length 20
00> [Config.cpp@249 ERROR]: Data set 2 moduleId : 536871936 Module version : 153 length 0
00> [Config.cpp@286 CONFIG]: No flash config for module 0x01F401F0 found, using defaults

I do see the module ID, version and length mismatch, but not sure why. Is it because of faulty print parameter. Can you help diagnose this please.

PriyeshNema commented 2 years ago

Seems like Module ID's address is getting printed. Is that the one getting compared also?

mariusheil commented 2 years ago

Does the example from the DebugModule documentation work for you?

//E.g. saves or updates a record with ID 1234 and data AA:BB:CC saverec 1234 AA:BB:CC

getrec 1234

If that does not work, it is not about the moduleIds or configuration length

PriyeshNema commented 2 years ago

If that is just about sending commands through RTT and checking, it does work

  < saverec 1234 AA:BB:CC
00> [RecordStorage.cpp@183 RS]: SaveRecord id 1234, len 3
00> [FlashStorage.cpp@124 FLASH]: Queue CachedWrite 537129184 to 1044484 (12)
00> [FlashStorage.cpp@197 FLASH]: processing command 2
00> [FlashStorage.cpp@248 FLASH]: copy cached data to 1044484, length 12
00> [FlashStorage.cpp@303 FLASH]: Flash operation success
  < getrec 1234
00> AA:BB:CC: (3)

I will have to directly call the functions in these commands and put in my code to check. Doing that.

PriyeshNema commented 2 years ago

Hi @mariusheil

For just testing purpose, I implemented a terminal command to save the data in flash. This command I send whenever the device is disconnected (my setup has just a peripheral and central for now, no mesh), and I call Saverecord() from the command and this works. During reboot, when I call GetRecord() in ConfigurationLoadedHandler() I'm able to read the record I saved.

This is kind of workaround as we don't want anyone to keep sending saverecord command from terminal in field. I'm curious how this can be achieved in FruityMesh environment where each device is always connected to 3-4 other devices. If writing to flash only works when no device is connected on BLE, it's very difficult to achieve in FruityMesh environment.

mariusheil commented 2 years ago

Hi, sorry for the late response. I can't really tell you at the moment why saving doesn't work for you while you are connected. Are you maybe storing a lot of data at once? I am not sure at the moment if the flash library tries to save it in chunks. I don't have the time to reproduce this at the moment, sorry. I think you need to do some experimenting. We do not experiance similar behavior with our nodes and we are perfectly able to store data in flash while the device is connected to other devices. It could be that you are doing it at a bad time, e.g. while some other high importance task is running. Try playing around with the number of retries in the flash library.

Marius

PriyeshNema commented 2 years ago

ok, no issues. As far as size is concerned, it's 20 bytes only and it's only 1 chunk and 1 time.

mariusheil commented 2 years ago

Hi,

that's weird. If you can send us something to reproduce the issue, we might be able to help (not sure on the timeline though). Otherwhise, I'll keep this issue open for now so that we can reproduce it when we have a bit of time. If you have any more information, e.g. about the exact timing when you try to store it, etc,... feel free to add it.

Marius

mariusheil commented 2 years ago

Hi, I'll close this for now, if there is anything new, don't hesitate to re-open it.

bluerange-io / bluerange-mesh

Fruity Mesh: How to save data in Flash #192