[cygnus-ngsi][OrionStatsSink] Create

[frbattid]

What about a sink receiving (certain) Orion notifications and (online) computing some kind of simple statistics for each entity's (numerical) attribute, in order to update new entities at Orion? This could the OrionStatsSink :)

More details:

• This sink will only work with a subset of the notifications, not all. This is because the statistics computing may result in a very time consuming task, requiring a lot of memory resources (although we'll try to avoid that point). Which notifications? This is clearly a new configuration parameter. Or be part of a global filtering system deciding which sink will receive each notification.
• In order to reduce the computation time and the resources usage, the sink must be able to compute the statistics online:
  • Maximum: if notified_value > max then max = notified_value. Please observe max=null at initial time, which denotes no value has been notified yet and the first notified value is directly assigned to max.
  • The same for the minimum.
  • The average and variation/standard deviation can be computed online as explained here and here, respectively.
  • See this link for other statiscal measures.
  • If sampling, maybe the distribution could be figured out.
• This should apply only to numerical attributes. Orion data types are merely descriptive, thus figuring out which attributes are numerical or not is a difficult task. Nevertheless, as previousl said, only those entity's attributes specified by the user will be used... thus let's hope the user chooses well :)

Regarding output entities to be updated in Orion, certain naming conventions should be used, e.g. if the entity's name is some_entity_name then the output entity could be named some_entity_name_stats. Other possibility could be to add the statistic values as metadata of the original entity's attribute.

In general, this is an online way of obtaining a subset of all the statistics that the Short-Term historic will be able to generate from previously stored batches.

[frbattid]

Since this is an experimental sink, not required by FIWARE nor the IoT Platform, I'll develop it in parallel to real backlog, using my free time ;)

[fgalan]

Good idea!

Orion implements an "/statistics" operation (see https://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/Publish/Subscribe_Broker_-_Orion_Context_Broker_-_Installation_and_Administration_Guide#Statistics) but this OrionStatsSink would be a very useful complement.

[frbattid]

What do you think about feeding Orion? Is it better to update a new "_stats" sufixed entity containing the statistics? Or could we take advantage of the metadata and add the statistics to the already existing entity?

[fgalan]

I really love the idea of using attribute metadata for attach statistics to Orion entities.

[frbattid]

Some resources from a previous trial; the following ones can be directly reused:

• OrionBackend interface --> https://github.com/telefonicaid/fiware-cygnus/commit/9519732d9cf8a30aed329ce441282f28672eb080
• OrionBackendImpl --> https://github.com/telefonicaid/fiware-cygnus/commit/774a9c3a5c6dfc52dc1a6511598bd40213b24ff7
• OrionBackendImplTest --> https://github.com/telefonicaid/fiware-cygnus/commit/932ff64ef023b57a1acba062f1f22d83ff29c8bb
• OrionStats --> https://github.com/telefonicaid/fiware-cygnus/commit/ce451e2fa77fb537cdae55c2bbd892091d0ce97f
• OrionStatsTest --> https://github.com/telefonicaid/fiware-cygnus/commit/ac005bd870a42a75932ed5e31dbe0856ca5a8ec4

This can be used as an inspiration of what must be done regarding the sink, but must be adapted to the latests changes in hte core (batches, data_model, etc):

• OrionStatsSink --> https://github.com/telefonicaid/fiware-cygnus/commit/687785624ba1dc4f8d61b01357ca7148e278d406