As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages

Luscha commented 1 year ago

As an Open Data Hub maintainer i want a precise message specification on top of which build and broadcast all messages.

Questions to be defined:

do we want to assign an ID to each message?
which fields do we want to have in each message (e.g. origin, data collector, timestamp, history, etc.)?

This is mainly a documentation issue.

sseppi commented 1 year ago

@Luscha can we discuss this issue a bit more into detail on Monday, in order to define the priority and the right Milestone?

clezag commented 4 months ago

The current format (by convention - as defined in WriterRoute) is

{
string provider
string timestamp
string rawdata
}

provider is in format provider/dataset
timestamp is in format ISO 8601 (not sure how timezones are handled - has to be tested)
rawdata is a string, could be a raw json or yaml, or base64 encoded whatever. max 16MB due to mongodb limitations

Reasonable candidates to be included are:

id (a unique ID propagated as correlation ID in rabbitmq, so that we can track a datapoint from start to finish)
content type to enable handling of specific data formats, and document what blobs actually are

noi-techpark / infrastructure-v2