Full filesystem due LittleFS

[ronnievdc]

From https://arduino-esp8266.readthedocs.io/en/latest/filesystem.html

LittleFS is recently added and focuses on higher performance and directory support, but has higher filesystem and per-file overhead (4K minimum vs. SPIFFS’ 256 byte minimum file allocation unit).

The logging.h module can create up to 57 logfiles if i'm correct. Each logfile taking at least 4KB of space, the filesystem can should need at least 228KB. Since it only has 128KB we can Revert back to SPIFFS or save more types of data into one file. For example

/HourE1.log
/HourE2.log
/HourG.log
/HourR1.log
/HourR2.log
/HourTE.log
/HourTR.log

could be saved into one file Hour.log with all the values of that hour. This could also make the logging code easier and less Write instructions to the flashdrive.

[romix123]

I am in favour of completely revamping the local data storage. The current configuration was made to allow for rapid graph production at the expense of storage efficiency. LittleFS was used instead of SPIFFS because SPIFSS stopped working at random moments. There still are many errors in the graph functions anyway.

Let's discuss requirements for graphs. Storing all core data (as listed above) in a single entry each hour and retaining the last one of the day for up to 13 months would allow all sorts of graphs to be produced.

[ronnievdc]

Ill suggest 3 files

• today.log (values for each hour today, starting at )
• yesterday.log (values for each hour yesterday, at today.log is renamed to yesterday.log)
• .log (values for each day of the year, starting at 1-1-xxxx, hour of logging: ). Each year a new file is created.

Dayrotation: time should be configurable to adjust for different timezones not counting DaylightSavingTime. Range: -12 to 12 hours.

Formatting of the line (i prefer Binary over ASCII):

struct StatisticsLineV1 {
  uint8_t version;     // starting with 1
  uint64_t timestamp;  // unix timestamp as returned by the Time now() method
  uint32_t energy_delivered_tariff1;  // total value in Watt (1000 x P1-value)
  uint32_t energy_delivered_tariff2;
  uint32_t energy_returned_tariff1;
  uint32_t energy_returned_tariff2;
  uint32_t gas_delivered;  // total value in dm3 (1000 x P1-value)
}

The version could be increased when we need the change the format of the line. Then its possible to check the first 8 bits to see which parser is needed for the rest of the line.

I think storing only the total values is needed. The difference can be calculated by comparing to the previous record.

[ronnievdc]

And some proposed class setup

// Handling saving data to the filesystem and retrieving data from the filesystem
class Statistics {
 public:
  explicit Statistics(FS &fileSystem) : mFileSystem(fileSystem) {}
  // rename today.log -> yesterday.log
  void rotateDay() {}
  // saves values in today.log
  void saveHour(const DSMRData &data) {}
  // saves values in <year>.log
  void saveday(const DSMRData &data) {}
  // helper method to create from/to timestamps
  static inline uint64_t createTime(uint8_t year, uint8_t month, uint8_t day) {
    tmElements_t tmElements = {0, 0, 0, 1, day, month, CalendarYrToTm(year)};
    return makeTime(tmElements);
  }
  // streams values of yesterday.log + today.log
  void getHours(const StatisticsStream &stream) {}
  // streams values of <year>.log in between from timestamp and to timestamp
  void getDays(const StatisticsStream &stream, const uint64_t from,
               const uint64_t to) {}
  // streams values of <year>.log in between from timestamp and to timestamp
  // but only 1st entry of the week
  void getWeeks(const StatisticsStream &stream, const uint64_t from,
                const uint64_t to) {}
  // streams values of <year>.log in between from timestamp and to timestamp
  // only 1st entry of the month
  void getMonths(const StatisticsStream &stream, const uint64_t from,
                const uint64_t to) {}

 private:
  FS &mFileSystem;
};

// Interface class handle lines from the logfiles
// I chose for streams because logfiles can become big
// therefore handling line by line seems a good idea to manage resources.
class StatisticsStream {
 public:
  StatisticsStream() {}
  virtual void parseLine(const StatisticsLineV1 &line) = 0;
};

// Sample class that prints the energy_delivered_tariff1 per line to the Serial port.
class StatisticsStreamSerial : public StatisticsStream {
 public:
  explicit StatisticsStreamSerial(HardwareSerial &serial) : mSerial(serial) {}
  void parseLine(const StatisticsLineV1 &line) override {
    mSerial.println(line.energy_delivered_tariff1);
  }

 private:
  HardwareSerial &mSerial;
};

///////////////////
// Example usage //
///////////////////
Statistics s(FST);
s.rotateDay();
s.saveHour(dsmrData);
s.saveday(dsmrData);

StatisticsStreamSerial ss(Serial);
s.getDays(ss, s.createTime(2024, 1, 1), s.createTime(2025, 2, 1));
s.getWeeks(ss, s.createTime(2024, 1, 1), s.createTime(2025, 12, 1));
s.getMonths(ss, s.createTime(2000, 1, 1), s.createTime(2030, 1, 1));

[romix123]

cool