search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
65 stars 105 forks source link

Bug Cygnus persist STH (or other sinks) data batch_size and batch_timeout #2045

Open cesarjorgemartinez opened 3 years ago

cesarjorgemartinez commented 3 years ago

Bug Cygnus persist STH (or other sinks) data batch_size and batch_timeout

Reading https://github.com/telefonicaid/fiware-cygnus/blob/master/doc/cygnus-ngsi/flume_extensions_catalogue/ngsi_sth_sink.md explain two parameters:

batch_size Mandatory no Default 1 => Number of events accumulated before persistence. batch_timeout Mandatory no Default 30 => Number of seconds the batch will be building before it is persisted as it is.

Then the osbserved behavihour is the following:

But must be:

Other sinks are affected?

Regards, Cesar Jorge

AlvaroVega commented 3 years ago

Current implementation is: https://github.com/telefonicaid/fiware-cygnus/blob/10f5751002d85ace0bf050af2379dc280c48ec71/cygnus-ngsi/src/main/java/com/telefonica/iot/cygnus/sinks/NGSISink.java#L516-L521

AlvaroVega commented 3 years ago

Each Cygnus sinks has their own propertires for

cygnus-ngsi.sinks..batch_size = 1 cygnus-ngsi.sinks..batch_timeout = 30

Regarding with Mongo typically there are two sinks (mongo-sink and sth-sink). Make sure both of then has these properties defined as you want.