Cache Controller[experimental]: Dumping cached data on request to file

[Wiki591]

Using the cache controller I noticed, that then and now the export of the chached data is missing the last up to 9 entries. I suppose, that these data sets are still in memory cache and not written to the files (looks like as if the cache controller caches ten sets of data in memory). In my current project sometimes there are the last steps of my travel missing. Would be nice, if there could be the possibility to force a write of these data into the corresponding files.

[TD-er]

The Cache controller does cache upto 240 bytes in RTC memory before writing them to the file system. This way the samples are not lost when the unit reboots before writing them to flash and also it puts less burden on the flash chips.

Since each sample takes 24 bytes, it keeps 10 samples in memory.

So as a quick dump, you could perform 9 calls of taskrun on a dummy connected to this controller.

I can see why such a command can be useful though.

[Wiki591]

I already got he opportunity to see that the cache controller data are surviving a reboot, well done, Sir.

And using TaskRun for dumping the data is a good idea, should have got it myself. The data have to be altered anyway and the additional informations are easily to identify and to remove. Thanks.

Btw.: Is there any plan to develop the .htm for extracting the data furthermore? I've seen in the comments that you are aware of some needs....

[TD-er]

There is a group of students currently working on adding the feature to ESPEasy to automatically uploading the data from the bin files as soon as WiFi has been restored. (I do assist them)

And I do have some test code in progress to parse the captured .CSV files in Python to process them in a database.

So there is even more than just a plan to do further work on this. The only thing missing is a clear deadline on when this will be ready.

[Wiki591]

Oh, wow, sounds good. For me personally no deadline needed, I want to get the data on demand anyway, absolutely no automation needed or wanted (I do have a network onboard, the networks mobile/home are strictly seperated).

[happytm]

I want to compile under PlatformIO - VSCode including cache controller. Can you describe the options I have to select in platformIO.ini file.

Thanks.

[TD-er]

It is easiest to build the 'custom' build and either edit the existing https://github.com/letscontrolit/ESPEasy/blob/mega/tools/pio/pre_custom_esp82xx.py Or rename the Custom-sample.h to Custom.h and make sure to define USES_C016

#define USES_C016

[happytm]

@TD-er Thank you.

[TD-er]

This is now also implemented using the new Cache Reader plugin. It can upload to MQTT either in binary format or as CSV. It can also generate a download CSV.

These CSV formats can be a bit more 'compressed' as it allows to combine samples with the same timestamp to the same line.

To allow more values to be saved to the Cache Controller using the same timestamp. one can append a unixtime to the new command TaskRunAt, which is currently only practical on sysinfo and dummy tasks as those don't need to actually interact with hardware.

Rules example:

On gps#All Do
  Let,4,%unixtime%
  LoopTimerSet,1,[int#2] // re-set the timer to have at least [int#2] seconds between samples.
  TaskValueSet,gps1,long,[gps#long] 
  TaskValueSet,gps1,lat,[gps#lat] 
  TaskValueSet,gps1,alt,[gps#alt] 
  TaskValueSet,gps1,spd,[gps#spd] 
  TaskRunAt,gps1,[int#4]
  Event,saveAllToFlash
Endon

On Rules#Timer=1 Do
  Let,4,%unixtime%
  Event,saveAllToFlash
Endon

On saveAllToFlash Do

   // Now collect  other data to store into 'dummy' tasks.
   // For this, we use the command TaskValueSet.
   // See: https://espeasy.readthedocs.io/en/latest/Rules/Rules.html#taskvalueset

   // First we get the other values from the GPS unit and store it  in the dummy task called "gps2"
   // See: https://espeasy.readthedocs.io/en/latest/Plugin/P082.html#access-to-all-measurement-values
   // N.B. the first 2 arguments of the TaskValueSet command are resp. the task name and variable name of a dummy task
   // So make sure these match with an enabled dummy task
   TaskValueSet,gps2,hdop,[gps#hdop] 
   TaskValueSet,gps2,satvis,[gps#sat_vis] 
   TaskValueSet,gps2,sattracked,[gps#sat_tr] 
   TaskValueSet,gps2,chksumfail,[gps#chksum_fail] 
   // Calling TaskRun will cause the values to be sent to the controller(s) assigned to the task
   TaskRunAt,gps2,[int#4]

   // Now we collect the latest samples of the analog voltages
   TaskValueSet,analog,carBat,[bat#Analog]
   TaskValueSet,analog,backupBat,[batBackup#Analog]
   // And flush them to the Cache Controller
   TaskRunAt,analog,[int#4]

   // Now we collect the latest samples of the BME280
   TaskValueSet,bme2,Temperature,[bme280#Temperature]
   TaskValueSet,bme2,Humidity,[bme280#Humidity]
   TaskValueSet,bme2,Pressure,[bme280#Pressure]
   // And flush them to the Cache Controller
   TaskRunAt,bme2,[int#4]

   // Store the samples from the last reading from the PMS5003
   TaskRunAt,pms1,[int#4]
   TaskRunAt,pms2,[int#4]

   // Collect data from the (optional) Accelerometer 
   TaskRunAt,adxl345,[int#4]

   // Collect some sysinfo
   TaskRunAt,sysinfo,[int#4]  

   // Collect error states
   TaskRunAt,error,[int#4]  
Endon

// Collect all values from the PMS5003 sensor
// and store them in 2 dummy tasks
On pms5003#pm1.0 Do
     TaskValueSet,pms1,pm1_0,%eventvalue1%
Endon

On pms5003#pm2.5 Do
     TaskValueSet,pms1,pm2_5,%eventvalue1%
Endon

On pms5003#pm10 Do
     TaskValueSet,pms1,pm10,%eventvalue1%
Endon

On pms5003#cnt5.0 Do
     TaskValueSet,pms1,cnt5_0,%eventvalue1%
Endon

On pms5003#cnt0.3 Do
     TaskValueSet,pms2,cnt0_3,%eventvalue1%
Endon

On pms5003#cnt0.5 Do
     TaskValueSet,pms2,cnt0_5,%eventvalue1%
Endon

On pms5003#cnt1.0 Do
     TaskValueSet,pms2,cnt1_0,%eventvalue1%
Endon

On pms5003#cnt2.5 Do
     TaskValueSet,pms2,cnt2_5,%eventvalue1%
Endon