ging / fiware-draco

The Draco Generic Enabler is an alternative data persistence mechanism for managing the history of context. It is based on Apache NiFi and is a dataflow system based on the concepts of flow-based programming. It supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic and also offers an intuitive graphical interface
https://fiware-draco.readthedocs.io/en/latest/
Apache License 2.0
20 stars 14 forks source link

Fiware Draco - Subscription works only for the first payload. Only the first payload is saved in historical mongodb #66

Closed danijelakrstic closed 2 years ago

danijelakrstic commented 2 years ago

I am trying to save historical context data in Mongo, but without success. Only the first payload sent to Draco is saved to MongoDB for historical data, but Mongo does not react to attribute updates. Versions used for the test: Orion-LD version 0.8.0 and Orion 3.4, Mongo version 4.4, Draco version 1.3.6. I tested it also with the 3.4 version of Mongo and the behavior is the same.
When I tested the processors for Mysql database there were no problems.

Can you, please, help me to fix a problem?

Below are the steps I performed:

Create a Draco subscription:

curl --location --request POST 'http://localhost:1026/v2/subscriptions' \
--header 'Fiware-Service: test' \
--header 'Fiware-ServicePath: /openiot' \
--header 'Content-Type: application/json' \
--data-raw '{
  "description": "Notify Draco of all context changes",
  "subject": {
    "entities": [
      {
        "idPattern": ".*"
      }
    ]
  },
  "notification": {
    "http": {
      "url": "http://10.0.0.5:5050/v2/notify"
    }
  },
  "throttling": 0
}'

Create an entity:

curl --location --request POST 'http://localhost:1026/v2/entities' \
--header 'Fiware-Service: test' \
--header 'Fiware-ServicePath: /openiot' \
--header 'Content-Type: application/json' \
--data-raw ' {
      "id":"urn:ngsi-ld:Product:0102", "type":"Product",
      "name":{"type":"Text", "value":"Lemonade"},
      "size":{"type":"Text", "value": "S"},
      "price":{"type":"Integer", "value": 99}
}'

Overwrite the value of an attribute value:

curl --location --request PUT 'http://localhost:1026/v2/entities/urn:ngsi-ld:Product:0102/attrs' \
--header 'Fiware-Service: test' \
--header 'Fiware-ServicePath: /openiot' \
--header 'Content-Type: application/json' \
--data-raw '{
    "price":{"type":"Integer", "value": 110}
}'

LISTEN_HTTP PROCESSOR: 1

NGSITOMONGO PROCESSOR: 2

Template: 3

MongoDB: mongo_db

YamacTan commented 2 years ago

I am having exactly the same problem in my project. There is no problem with my data updates on Orion. In addition, I also see that Orion sends the necessary subscription payload to Draco with every update.

When I persist the same data on MongoDB using STH Comet, I get the result I want. There may be a problem with Draco itself or with the NGSItoMongo processor. I'll be following for issue updates.

anmunoz commented 2 years ago

It seems to be a bug, from the Draco team we are going to check this issue and try to solve it as soon as possible.

danijelakrstic commented 2 years ago

Please let us know of any changes. We should use Draco in the project, so we have to find a solution as soon as possible. Thank you very much in advance.

anmunoz commented 2 years ago

Hi @danijelakrstic

In the new version of Draco 2.1.0 this bug is fixed. You can check the code in the official repository. The release link is https://github.com/ging/fiware-draco/releases/tag/2.1.0

Additionally, you can use the docker image available for this release by pulling it using docker pull ging/fiware-draco:2.1.0.

You can also use the Mongo-Tutorial template available inside of Draco where you have preconfigured the processors needed to persist in MongoDB.

One thing that you have to consider is that the new version of Draco is aligned with the 1.15.3 version of NiFi in where you need first to log in for access to the Web UI using the default credentials (user: admin, password: pass1234567890). You can check the official documentation for more information about it https://fiware-draco.readthedocs.io/en/latest/

danijelakrstic commented 2 years ago

Thank you for your work. Great!