search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 104 forks source link

Remove guard for initial notifications #2044

Open fgalan opened 3 years ago

fgalan commented 3 years ago

Initial notifications are going to be deprecated in Orion Context Broker. They are being deprecated in Orion 3.1.0 and the removal is expected for 3.2.0.

When that moment comes, the Cygnus guard code to "protect" against initial notification will no be longer needed.

This PR can be an useful reference: https://github.com/telefonicaid/fiware-cygnus/pull/2043

fgalan commented 3 years ago

Similar to issue https://github.com/telefonicaid/perseo-fe/issues/526

jaimeventura commented 3 years ago

Hey @fgalan, I found this issue after hitting a problem with a initial notification with around 100k entities (i would love to talk to you about this number, but thats another subject.)

Anyway, I came across the issue on orion, about Initial notifications being deprecated, and since its mentioned here that "if you need to know the status of your system at subscription time, then use GET /v2/entities with proper pagination." i have a doubt.

I understand the reasons for deprecating this feature. But, for instance, when using cygnus with the Postgresql sink, how one can load every entities prior to the subscription? And, say 'all' entities are previously loaded to that database, how may one guarantee that there where no changes between the end of the loading process and the time the subscription was issued?

(Or maybe im misunderstanding the use of cygnus?)

fgalan commented 3 years ago

I found this issue after hitting a problem with a initial notification with around 100k entities (i would love to talk to you about this number, but thats another subject.)

I understand that 100k refers to the payload size of the entities, no the number of the entities. Initial notification always included as much as 20 entities (see https://github.com/telefonicaid/fiware-orion/issues/591).

However, initial notification feature has been already removed in master (so in next version, Orion 3.2.0 will not be) so talking about this is probably not needed :)

Anyway, I came across the issue on orion, about Initial notifications being deprecated, and since its mentioned here that "if you need to know the status of your system at subscription time, then use GET /v2/entities with proper pagination." i have a doubt.

I understand the reasons for deprecating this feature. But, for instance, when using cygnus with the Postgresql sink, how one can load every entities prior to the subscription? And, say 'all' entities are previously loaded to that database, how may one guarantee that there where no changes between the end of the loading process and the time the subscription was issued?

(Or maybe im misunderstanding the use of cygnus?)

The usual usage case in Cygnus is to persist history, i.e. changes in the entities. The initial status of the system is not considered history, from the point of view of Cygnus, only the changes since the feeding subscription is created. In fact, current Cygnus implementation discard the initial notification and pay attention only to regular (non-initial) notifications.

However, if you need to set an "initial change" in your storing system (Postgresql in your case) for every entity in your system, you have two alternatives:

  1. Pre-create that entry in the storing system, using some out-of-band mechanism. For instance, a script that reads entities for CB and insert in Postgresql Cygnus tables accordingly
  2. Trigger a notification with the initial value. You don't need to change the entities, you can use the ?options=forcedUpdate (described here) with the current value of some triggering attribute in the subscription.