telemetry/reporting support

[petersilva]

Each data source typically likes to be aware of where their data has travelled (who has consumed it.) and if there were difficulties in processing it. Whenever an item is downloaded, one could provide a telemetry report back to the source to indicate how processing went.

In Sarracenia, this is implemented as the same format of messages, but travelling in the opposite direction (from the consumer back to the source, perhaps not by the same route.) Differences:

• instead of v03.post, the topic of a telemetry report is v03.report

• content header is removed.

• a report header is added, as an object:

  • elapsedTime: the reported propagation delay based on the injection time received.
  • resultCode: an httpd based numeric code indicating how it went.
  • host: hostname submitting the report.
  • user: broker username submitting the report.

• from_cluster is used to indicate where the report should end up. Each implementation will need to determine what node to send a report to in order to get it closer to from_cluster

In Sarracenia, reports are not yet deployed, as there is a fundamental problem that the number of reports will dwarf the number of posts (10:1 is a typical estimate), and so likely separate infrastructure needs to be established to avoid overloading the data delivery one. This emphasizes that the routing of telemetry will likely be different that the post messages.

[petersilva]

suggest that v02 from_cluster would be more descriptively named in v03: reportsTo

[petersilva]

https://github.com/MetPX/sarracenia/issues/176

[petersilva]

None of this is implemented in the wmo_mesh at the moment.