Discussion: Observation block and individual data feeds

[kwilcox]

The current mocked sensor output groups many quantities into one block that I'll call the "observation" block. Q1: Will most individual parameters from a single sensor have the same reporting period/frequency? If so this works well and will save space/MQTT traffic when reporting from the sensors.

{
  "time": "2022-04-01T22:36:50Z",
  "latitude": 9.906,
  "longitude": -150.028,
  "altitude": 91.899,
  "temperature": 27.166,
  "rh": 61.317,
  "wind_speed": 7.282,
  "wind_dir": 104.854
}

We may want to translate the above into individual sensor feeds after reading from MQTT and push the values into a new MQTT topic. Separating out each individual feed will help organize data about that individual feed, such as QC results. We could use the information in the attributes header to generate unique feed identifiers for each sensor. That unique feed ID is extremely useful for going back and referencing the individual parameter coming off the sensor. The resulting feed messages from the above observation block would be:

{
  "feed": "/sensor/MockCo/Sensor_1/1234/temperature",
  "time": "2022-04-01T22:36:50Z",    # Always UTC
  "latitude": 9.906,                 # WGS84 latitude
  "longitude": -150.028,             # WGS84 longitude
  "z": 91.899,                       # meters positive up from WGS84
  "value": 27.166                    # measurement value
}

{
  "feed": "/sensor/MockCo/Sensor_1/1234/rh",
  "time": "2022-04-01T22:36:50Z",
  "latitude": 9.906,
  "longitude": -150.028,
  "z": 91.899,
  "value": 61.317
}

{
  "feed": "/sensor/MockCo/Sensor_1/1234/wind_speed",
  "time": "2022-04-01T22:36:50Z", 
  "latitude": 9.906,
  "longitude": -150.028,
  "z": 91.899,
  "value": 7.282,
}

{
  "feed": "/sensor/MockCo/Sensor_1/1234/wind_dir",
  "time": "2022-04-01T22:36:50Z",
  "latitude": 9.906,
  "longitude": -150.028,
  "z": 91.899,
  "value": 104.854,
}

[derekcoffman]

I had been thinking along these lines but wasn't sure exactly where to separate "observation block" from individual measurements. One thing I think we need to do is have a pathway that saves each block to ERDDAP as our level 0 (or whatever we will call it) data. But that does not preclude a second pathway breaking the data into individual measurements to start processing them.

To answer the question: Q1: Will most individual parameters from a single sensor have the same reporting period/frequency?

Answer: I would presume so. For our instruments, I would expect a complete data record (observation block) at a set frequency. Each instrument may report at a different frequency but with complete records. I can't say for certain this is the case for all the data this might cover but, for now, let's assume it does.

[kwilcox]

Should the ERDDAP datasets represent a sensor package or something higher level... maybe a "station" that has many sensor packages?

[derekcoffman]

I think that is up for discussion and may be a group by group decision. For our group's needs, I am looking at keeping things at the individual sensor level. Our data has, historically, been partitioned by field project where we have the same types of measurements but the instruments themselves get swapped around at times so a given instrument/sensor my be measuring ambient particle concentration (e.g.) but might get moved behind a thermal denuder which modifies the actual measurement characteristics. So, one instrument parameter might be the source of multiple project level variables for the same cruise. The idea would be to have log the maps instrument-> variable at any given time. This would mean a second ERDDAP dataset for the cruise data but keep the sensor data intact. Still working on this idea,though.

Other groups might not want/need that level of granularity.

On Mon, Apr 11, 2022 at 9:09 AM Kyle Wilcox @.***> wrote:

Should the ERDDAP datasets represent a sensor package or something higher level... maybe a "station" that has many sensor packages?

— Reply to this email directly, view it on GitHub https://github.com/derekcoffman/iot-data-landing/issues/5#issuecomment-1095253222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFFQ3ILOQ34R2V2EDWKUZDTVERFDZANCNFSM5TDS27JA . You are receiving this because you commented.Message ID: @.***>

--

---

Derek Coffman @.*** NOAA/PMEL phone: 206-526-6574 7600 Sand Point Way NE Seattle, WA 98115 fax: 206-526-6744

http://saga.pmel.noaa.gov

---

[derekcoffman]

Added a little more about this topic in here. Some of the doc is beyond the scope of what we are working on together but the Data Source section reiterates some ideas from above.