New feature: save readings for which the upload failed and retry on network reconnection

[dblachut]

Some people (including me) may have low Wi-fi signal strength in the place where their NAM device is placed. In such case it often happens that the reading is not uploaded to the servers. It would be great if the NAM s/w would preserve last N readings and upload them later with the timestamp of occurrence.

[awbnet]

You probably want to save it on flash which I think could wearout too fast? Depending on how bad your WiFi signal is.

[dblachut]

No, I was thinking about RAM since it should be few integers/floats. This would not make sense with network watchdog enabled though.

[awbnet]

Maybe it would make more sense if there was a module with SD card. NAM could write all measurements to it and if there is WiFi available again, push them online and erase from SD card. That would do for aqi.eco. The measurements (in JSON format) have timestamp, so they can be uploaded as bulk data and still be presented correctly.

[dblachut]

Yeah, but is SD card supported currently?

[awbnet]

Yeah, but is SD card supported currently?

I don't think so. It is up to @netmaniac to decide, wheter it is a good idea.

[netmaniac]

First - question is if API can accommodate time set in JSON. For sensors with GPS, time from GPS is being sent in JSON. But I don't know what API (Sensor Community, Madavi etc) will take as measurement timestamp - time from JSON or real receive time. If the second option, then probably we can not do anything about lost readings. Second - we could store few readings in memory (using binary format) to send it back later (if timestamp from JSON will be accepted by API), but not many of them. Flash chips have low guaranteed write cycles. Some time ago we were considering storing "lost" measurements as JSON in SPIFFS, but it looked like we can get into some problems relatively quick.

Ofc - saving data will make sense when we lost connection (and NAM has time set from NTP) and for not long (need to calculate how many data can be safely stored).

More sophisticated diagnostic data collection service is being developed now. When it's done (mid Dec) then we can get more data from sensors. I will try to measure how often sensor is not getting response OK from API. Then I will decide if storing data for later is something useful.

[dblachut]

First - question is if API can accommodate time set in JSON.

I think I have mistakenly assumed that AQI is taking timestamp from JSON but after some analysis I think I was wrong. I will consult this with @trekawek, even if I am right I think he will be open to change it.

Second - we could store few readings in memory (using binary format).

So your suggestion is to store the whole data that is uploaded to server? I think it would be better (less memory consuming) to keep just the readings and built JSON on demand. Is this hard to achieve in current implementation?

I will try to measure how often sensor is not getting response OK from API.

Thanks but assuming the server is working correctly this will depend on the local environment so it might be different for each user. My assumption was that usually the NAM is placed outside the apartment where the Wi-Fi signal is poor and some readings may fail to be uploaded.

[trekawek]

Yes, I think I have mistakenly assumed that AQI is taking timestamp from JSON but after some analysis I think I was wrong. I will consult this with @trekawek, even if I am right I think he will be open to change it.

If the GPS timestamp is present in the JSON, aqi.eco will use it: https://github.com/trekawek/air-quality-info/blob/bf336feb9aa65d1a69ea0d3bca32f887e23f6df1/src/htdocs/model/updater.php#L42-L44

Otherwise, the server time will be used: https://github.com/trekawek/air-quality-info/blob/5765cd25aec23d7f94e12898f13fe8de22383854/src/htdocs/api/controller/update_controller.php#L27 https://github.com/trekawek/air-quality-info/blob/5765cd25aec23d7f94e12898f13fe8de22383854/src/htdocs/job/update_job.php#L36

[dblachut]

Thanks! Do you see any potential problems if AQI would start to use timestamp from JSON? What do you think about such idea?

[trekawek]

Thanks! Do you see any potential problems if AQI would start to use timestamp from JSON? What do you think about such idea?

Sure, that's a good idea. For now I see no timestamp in the JSON, maybe it's a matter of my firmware. Could you share an example of JSON with the timestamp?

[dblachut]

I have done some further research and it looks like namf is not sending timestamp in JSON. I don't know why I assumed it does but I guess it will be harmless to add it in the following way:

{
   "esp8266id":"0011223344",
   "software_version":"NAMF-2020-39a7",
   "timestamp": 1637923684470,
   "sensordatavalues":[
      {
         "value_type":"BME280_temperature",
         "value":"8.20"
      },
      {
         "value_type":"BME280_humidity",
         "value":"74.68"
      },
      {
         "value_type":"BME280_pressure",
         "value":"102182.70"
      },
      {
         "value_type":"SDS_P1",
         "value":"41.70"
      },
      {
         "value_type":"SDS_P2",
         "value":"26.50"
      },
      {
         "value_type":"HECA_temperature",
         "value":"19.87"
      },
      {
         "value_type":"HECA_humidity",
         "value":"31.63"
      }
   ]
}

I have basically added new field named timestamp, timestamp_ms could be an alternative name.

[trekawek]

@dblachut, sure, aqi.eco can use this field as soon as it starts appearing.

[dblachut]

Thanks once again, so I guess now it is up to @netmaniac to decide whether this feature will be supported.

[awbnet]

@dblachut So I guess timestamp should be used when device has synchronized its time from NTP server right? Otherwise it makes no sense. What if synchro fails and NAM won't have proper time? Timestap shouldn't be used then right?

[dblachut]

Fair point, I haven't thought of it.

[awbnet]

So timestamp could be used when NTP is synchronized.

Another thing for NTP synchro – how NAM performs it currently? Only once on bootup? If so, maybe it could perform another try after last try fails? And so on until it will synchronize time.