Lightweight ingestion (updates)

telefonicaid / fiware-orion

Context Broker and CEF building block for context data management, providing NGSI interfaces.

https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md

GNU Affero General Public License v3.0

211 stars 265 forks source link

Lightweight ingestion (updates) #3003

Open fgalan opened 7 years ago

fgalan commented 7 years ago

Current IOTAgent + CB deployments rely in HTTP comunication to update active attributes from IOTAgent to CB. Given that the load of data that IOTAgent may update can be huge (e.g. big cities with thousands of sensors reporting measures with high frequency) this is a critical data ingestion path that should be optimized with lighter mechanisms that HTTP.

Thus, several ideas in this line:

To use lightweight payloay serialization (e.g. REDIS-like)
To use binary communications, e.g. using Avro (https://avro.apache.org/), also mentioned as a solution for the CB->Cygnus optimization in issue #3002

kzangeli commented 7 years ago

Another option would be to implement our own protocol (binary, of course) for this, tailor-made for its purpose. Pretty easy to implement in the broker, not sure how hard it would be to in Cygnus an its alike. IMHO, internal communication should be made as fast as possible, i.e. using a tailor-made protocol.

fgalan commented 7 years ago

Tailor-made options will make more difficult third party software integrations. Avro seems to have a good balance between efficiency and third party integration (in addition, it is aligned with Cygnus roadmap, see https://github.com/telefonicaid/fiware-cygnus/issues/1315), although it would be great to have more feedback about possible alternatives.