Query timing changes data reported

[patricktokeeffe]

The 5-min offset for data retrieval, combined with only hourly updates causes RPi-Monitor to interpolate the Status values, which makes decoding the flags very difficult.

Not sure how long the EBAM takes to generate an hourly report, but probably <5min.

To complicate data retrieval timing, the EBAM clock drifts independently of the Pi and needs to be accomodated.

[patricktokeeffe]

Timing is hard here: to obtain "correct" timestamps, data must be obtained from EBAM, written to file, and queried by RPi-Monitor within the first minute of each hour.

To consider:

• unknown synchronicity between EBAM & RPi clocks
• unknown data record generation time of EBAM
• query rate/offset of the RPi-Monitor service

Current plan:

• establish new cron-based service to keep EBAM clock synced to Pi
• change hourly DAQ script to query for new records, not just the last record
  • launch at top of each hour (hopefully to finish within first minute)
  • retry several times until new record obtained
• (possibly) change RPi-Monitor query rate for EBAM data 10s → 1min?
  • may require 10s query rate to prevent 0:00-data from getting 0:01 timestamp
  • may require 60s query rate to prevent inter-minute averaging of values (which then become nonsense values)

[patricktokeeffe]

Examples of the timing mismatch producing nonsense values: -32.195 in this case, where it should be either 4 or -99. (XREF https://github.com/XavierBerger/RPi-Monitor/issues/276)

• unclear why status isn't updated until after first minute of hour (EBAM or python script?)
  • cron currently triggered at minute 1, not minute 0 - will fix!
• interpolation at point 10:01:10 is not linear between adjacent ports (why?)
• note RPi-Monitor query timing is aligned to zero-seconds (e.g. :00, :10, :20, :30, ...)
  • assuming EBAM & computer have same clock time, it's not possible to keep RPi-Monitor from including previous data in new minute's average (:00 will always hold old data)

[patricktokeeffe]

• unclear why status isn't updated until after first minute of hour (EBAM or python script?)
  • cron currently triggered at minute 1, not minute 0 - will fix!

Calling data script at minute 0 instead of 1, with 5 second query delay, does not produce latest data record because of EBAM clock offset (10:59:49 vs 11:00:05 = 16sec behind):

lar@ebam-plus:~/ebam-plus $ ls -l /tmp
total 16
-rw-r--r-- 1 root root  199 Jul 12 11:00 ebam.csv

tail -n 10 /home/lar/ebam-plus/src/getbam.log

Data Report
2019-07-12 10:59:49
Location, 3, W15096

Time,ConcRT(ug/m3),ConcHR(ug/m3),Flow(lpm),WS(m/s),WD(Deg),AT(C),RH(%),BP(mmHg),FT(C),FRH(%),BV(V),PM,Status
2019-07-12 10:00:00,+031663,+099999,+11.6,00.0,000,+024.5,041,694,+033.2,017,12.5,0,00512
4
gooddata = 1
New hour saved successfully.

---

Best (?) solution: set EBAM clock to intentionally run ~10sec ahead of RPi, so that data record is ready by start of new minute (xref #9)

• prevent :00 values from holding-over old data
• maintains sufficient time-synchronicity (i.e. constant 10s lead time is totally OK for hourly data sets)
• does not pollute :40 or :50 values because data queried at most minutely

[patricktokeeffe]

Still seeing strange averaging after start of each hour:

Try setting EBAM 30sec ahead of the Pi, and test RPi-Monitor at both 30s and 60s query rates.