Problem to notify Quantunleap

alvaroo278 commented 1 year ago

Description Issue I am trying to send a notification from Orion-LD to Quantumleap, but nothing arrives in the QL service. To check that Orion is notifying correctly, I am using a duplicate subscription that has as an endpoint a service echo (Webhook). I am using Kubernetes (ArgoCD) to configure and control the service.

Orion-LD configuration

apiVersion: v1
kind: Service
metadata:
  labels:
    app: orion
  name: orion
spec:
  ports:
  - name: http
    port: 1026
    protocol: TCP
    targetPort: 1026
  selector:
    app: orion

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orion
spec:
  replicas: 1
  selector:
    matchLabels:
      app: orion
  template:
    metadata:
      labels:
        app: orion
    spec:
      containers:
        - command: ["orionld"]
          args: ["-fg", "-multiservice", "-ngsiv1Autocast", "-dbhost", "mongodb",
               "-logLevel", "DEBUG"]
          image: "fiware/orion-ld:1.0.1"
          imagePullPolicy: IfNotPresent
          name: orion
          ports:
          - containerPort: 1026
            name: http

Quantumleap configuration

apiVersion: v1
kind: Service
metadata:
  labels:
    app: quantumleap
  name: quantumleap
spec:
  ports:
  - name: http
    port: 8668
    protocol: TCP
    targetPort: 8668
  selector:
    app: quantumleap

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: quantumleap
  labels:
    app: quantumleap
  annotations:
    # Tell Reloader to bounce the Argo CD server whenever the SSO config
    # and built-in admin credentials change.
    secret.reloader.stakater.com/reload: "postgres-users"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: quantumleap
  template:
    metadata:
      labels:
        app: quantumleap
    spec:
      initContainers:
        - name: quantumleap-pg-init
          image: "smartsdk/quantumleap-pg-init:0.7.6"
          imagePullPolicy: IfNotPresent
          env:
          - name: PG_HOST
            value: postgres
          - name: PG_PASS
            valueFrom:
              secretKeyRef:
                name: postgres-users
                key: postgres.password
          - name: QL_DB_PASS
            valueFrom:
              secretKeyRef:
                name: postgres-users
                key: quantumleap.password
          - name: QL_DB_INIT_DIR
            value: /db-init
          volumeMounts:
          - name: db-init
            mountPath: /db-init
      containers:
        - image: "orchestracities/quantumleap:0.8.3"
          imagePullPolicy: IfNotPresent
          name: quantumleap          
          ports:
          - containerPort: 8668
            name: http
          env:
          - name: POSTGRES_HOST
            value: "postgres"
          - name: POSTGRES_PORT
            value: "5432"
          - name: POSTGRES_USE_SSL
            value: "False"
          - name: POSTGRES_DB_NAME
            value: "quantumleap"
          - name: POSTGRES_DB_USER
            value: "quantumleap"
          - name: POSTGRES_DB_PASS
            valueFrom:
              secretKeyRef:
                name: postgres-users
                key: quantumleap.password
          - name: CRATE_HOST
            value: "crate"
          - name: CRATE_PORT
            value: "4200"
          - name: QL_CONFIG
            value: /config/db-routing.yaml
          - name: DEBUG
            value: "true"
          volumeMounts:
          - name: db-routing
            mountPath: /config
      volumes:
        - name: db-routing
          configMap:
            name: db-routing
        - name: db-init
          emptyDir: {}

Regarding the quantumleap images I use, if I use as initializer the orchestracities image (whatever version I use), the pod doesn't start.

Orion-LD subscription

{
    "id": "urn:ngsi-ld:Subscription:63dbab3998657c37179501a9",
    "type": "Subscription",
    "description": "Notify alert",
    "entities": [
      {
        "type": "AirQualityAlert"
      }
    ],
    "notification": {
      "attributes": [
        "analysis",
        "dateIssued",
        "prediction"
      ],
      "format": "normalized",
      "endpoint": {
        "uri": "http://192.168.1.18/quantumleap/v2/notify",
        "accept": "application/json"
      }
    },
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
  }

On the other hand, if I enter the logs of the quantumleap pod, it appears empty: ``` kubectl logs -f {id-pod}````

I have also tried to manually notify Quantumleap and I get a timeout error when making the request

Post Request

curl --location --request POST 'http://192.168.1.18/quantumleap/v2/notify' \
--header 'ngsild-tenant: airqualityalert' \
--header 'fiware-servicepath: /' \
--header 'Content-Type: application/json' \
--data-raw '{
  "id": "urn:ngsi-ld:Notification:63e274e0cb0b200efafa99ee",
  "type": "Notification",
  "subscriptionId": "urn:ngsi-ld:Subscription:63dbaa4998657c37179501a3",
  "notifiedAt": "2023-02-07T15:57:20.549Z",
  "data": [
    {
      "id": "urn:ngsi-ld:AirQualityAlert:Out.3857499894.1",
      "type": "AirQualityAlert",
      "analysis": {
        "type": "Property",
        "value": "good"
      },
      "prediction": {
        "type": "Property",
        "value": "good"
      },
      "dateIssued": {
        "type": "Property",
        "value": {
          "type": "DateTime",
          "value": "2023-02-07T15:57:20.519Z"
        }
      }
    }
  ]
}'

alvaroo278 commented 1 year ago

All pods are running well.

chicco785 commented 1 year ago

Regarding the quantumleap images I use, if I use as initializer the orchestracities image (whatever version I use), the pod doesn't start.

that's something @c0c0n3 needs to look into.

Orion-LD subscription

{
    "id": "urn:ngsi-ld:Subscription:63dbab3998657c37179501a9",
    "type": "Subscription",
    "description": "Notify alert",
    "entities": [
      {
        "type": "AirQualityAlert"
      }
    ],
    "notification": {
      "attributes": [
        "analysis",
        "dateIssued",
        "prediction"
      ],
      "format": "normalized",
      "endpoint": {
        "uri": "http://192.168.1.18/quantumleap/v2/notify",
        "accept": "application/json"
      }
    },
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
  }

what's this url "http://192.168.1.18/quantumleap/v2/notify ?

unless this is something you installed, you won't find quantum leap there. 1) the path it's just /v2/notify 2) you better use hostnames and not ips.

c0c0n3 commented 1 year ago

hi @alvaroo278 :-) It looks like you're using QL inside the Kitt4sme demo cluster? In that case, the URL should be

http://quantumleap/v2/notify

Can you give it a try?

alvaroo278 commented 1 year ago

/quantumleap it's equal :8668. I'm checked

I checked if Orion-LD can connect with QL. If I execute a Get request inside of pod Orion-LD and I receive version of QL.

[root@orion-66b4c55bc5-gndw9 /]# curl --location --request GET 'http://192.168.1.18/quantumleap/version'
{
  "version": "0.8.3-dev"
}

c0c0n3 commented 1 year ago

keep in mind you're getting out of the cluster and then routed back in through the Istio gateway, so it's best to make an internal call w/ the url I gave you earlier. The reason that doesn't work is that probably orion gets a 403 from the Istio gateway---security rules if memory serves. If you try any other URL from the Orion pod you should get a 403 too. The reason why the version endpoint works is that that url is excluded from the security check

https://github.com/c0c0n3/kitt4sme.live/blob/main/deployment/mesh-infra/security/ingress-policy.yaml#L31

alvaroo278 commented 1 year ago

I changed uri in the subscription, but nothing happened. My stream is FogFlow (ngsild entity) -> Orion-LD -> Quantumleap. So, it's necessary to put any header in the subscription or something? Because if I remember, I filled in this file:

https://github.com/c0c0n3/kitt4sme.live/blob/main/deployment/mesh-infra/security/secrets/templates/postgres-users.yaml

With a personal token and push in to my repo.

If I update the entity, I receive this in my echo service:

But If I check the entity in QL with the next curl:

curl --location --request GET 'http://192.168.1.18:8668/v2/entities/urn:ngsi-ld:AirQualityAlert:Out.3857499894.1' \
--header 'ngsild-tenant: airqualityalert'

I don't receive errors code but the answer is empty. All request were find in:

https://www.postman.com/fiware/workspace/fiware-foundation-ev-s-public-workspace/collection/513743-86151708-fb9d-42f2-a2fe-e40b4ea38861?ctx=documentation

But it's weird because I can't see anything inside logs quantumleap. I may make a mistake checking entities in QL.

New subscription

curl --location --request POST 'http://192.168.1.18/orion/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/json' \
--header 'ngsild-tenant: airqualityalert' \
--data-raw '{
    "description": "Notify alert",
    "type": "Subscription",
    "entities": [
        {
            "type": "AirQualityAlert"
        }
    ],
    "notification": {
        "attributes": [
            "analysis",
            "dateIssued",
            "prediction"
        ],
        "format": "normalized",
        "endpoint": {
            "uri": "http://quantumleap/v2/notify",
            "accept": "application/json"
        }
    }
}'

Request subscriptions

curl --location --request GET 'http://192.168.1.18/orion/ngsi-ld/v1/subscriptions/' \
--header 'fiware-service: airqualityalert' \
--header 'fiware-servicepath: /airqualityalert'

Response

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   814  100   814    0     0   262k      0 --:--:-- --:--:-- --:--:--  397k
[
  {
    "id": "urn:ngsi-ld:Subscription:63dbaa4998657c37179501a3",
    "type": "Subscription",
    "description": "Notify alert",
    "entities": [
      {
        "type": "AirQualityAlert"
      }
    ],
    "notification": {
      "attributes": [
        "analysis",
        "dateIssued",
        "prediction"
      ],
      "format": "normalized",
      "endpoint": {
        "uri": "https://webhook.site/d18d4b5d-d40a-4a71-a714-c26acf7aa7be",
        "accept": "application/json"
      }
    },
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
  },
  {
    "id": "urn:ngsi-ld:Subscription:63e36541cb0b200efafa9a04",
    "type": "Subscription",
    "description": "Notify alert",
    "entities": [
      {
        "type": "AirQualityAlert"
      }
    ],
    "notification": {
      "attributes": [
        "analysis",
        "dateIssued",
        "prediction"
      ],
      "format": "normalized",
      "endpoint": {
        "uri": "http://quantumleap/v2/notify",
        "accept": "application/json"
      }
    },
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld"
  }
]

All requests are sent inside the computer where the service is hosted.

If it's necessary, you can check all code of deployment inside my repo.

https://github.com/alvaroo278/kitt4sme.live/tree/main/deployment

alvaroo278 commented 1 year ago

hi @alvaroo278 :-) It looks like you're using QL inside the Kitt4sme demo cluster?

Hi, yes, I'm working on project KITT4SME and deploying the connexion FogFlow -> KITT4SME instance.

c0c0n3 commented 1 year ago

@alvaroo278 thanks so much for all the info! I'll try reproducing the scenario and see what breaks...

alvaroo278 commented 1 year ago

http://quantumleap/v2/notify

I just check how to create a subscription and then wrote:

http://quantumleap:8668/v2/notify (https://www.postman.com/fiware/workspace/fiware-foundation-ev-s-public-workspace/request/513743-b355261f-7dee-44db-aba3-b6b5499975de)

alvaroo278 commented 1 year ago

thanks so much for all the info! I'll try reproducing the scenario and see what breaks...

In my service (FogFlow), I introduce directly entity to json valid to Orion-LD, so I jumped Agents and sent my entity directly to Orion:

'http://192.168.1.18/orion/ngsi-ld/v1/entityOperations/upsert'

c0c0n3 commented 1 year ago

Hi @alvaroo278,

I think I finally got to the bottom of this. It looks like it's a configuration issue rather than a bug---thank goodness for that :-) Try reconfiguring your system like this

QuantumLeap's URL in the Orion subscription should be: http://quantumleap:8668/v2/notify. That's b/c inside your cluster you have a K8s service called quantumleap running on port 8668 which sits in the same namespace as Orion. (Yes, the URL I suggested earlier was wrong b/c it didn't have the port in it, sorry about that!)
You should tell QuantumLeap which DB backend to use for your tenant. If you want all your tenants to store data in Postgres, then add this env to your K8s deployment for QuantumLeap: QL_DEFAULT_DB=timescale. On the other hand, if you'd only like the airqualityalert tenant to have its data in Postgres, configure per-tenant routing.
Add an additional fiware-service: airqualityalert header to your subscription and upsert Orion calls. This is b/c at the moment Quantum Leap still doesn't support the ngsild-tenant header. But this feature should be merged into the mainline soon, see #695.

I couldn't manage to reproduce the container init issue. The container init image starts correctly on my side, it creates the Quantum Leap DB, and then exits.

c0c0n3 commented 1 year ago

@alvaroo278 here's a Docker Compose file you can use to test the points I mentioned above.

version: '3'

services:

  mongodb:
    image: mongo:4.4
    networks:
      - qltest

  orion:
    image: fiware/orion-ld:0.8.0
    entrypoint: orionld -fg -multiservice -ngsiv1Autocast -dbhost mongodb -logLevel DEBUG
    networks:
      - qltest
    ports:
      - "1026:1026"
    depends_on:
      - mongodb

  postgres:
    image: timescale/timescaledb-postgis:2.3.0-pg13
    ports:
      - "5432:5432"
    networks:
      - qltest
    environment:
      - POSTGRES_PASSWORD=*

  quantumleap-db-setup:
    image: smartsdk/quantumleap-pg-init:0.7.6
    depends_on:
      - postgres
    networks:
      - qltest
    environment:
      - QL_DB_PASS=*
      - QL_DB_INIT_DIR=/ql-db-init
      - PG_HOST=postgres
      - PG_PASS=*

  quantumleap:
    image: orchestracities/quantumleap
    depends_on:
      - postgres
    networks:
      - qltest
    ports:
      - "8668:8668"
    environment:
      - QL_DEFAULT_DB=timescale
      - POSTGRES_HOST=postgres
      - POSTGRES_PORT=5432
      - POSTGRES_DB_NAME=quantumleap
      - POSTGRES_DB_USER=quantumleap
      - POSTGRES_DB_PASS=*
      - POSTGRES_USE_SSL=False
      - USE_GEOCODING=False
      - CACHE_QUERIES=False
      - LOGLEVEL=ERROR

networks:
    qltest:
        driver: bridge

c0c0n3 commented 1 year ago

Start Docker Compose with the file above

$ docker compose up -d

Then when all services are up and running and the QL container init has exited, you should be able to get into the QL DB

$ psql postgres://quantumleap:*@localhost
psql (13.4, server 13.3)
Type "help" for help.

quantumleap=> \q

At this point you can set up your subscription with Orion like this

curl 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
    -H 'Content-Type: application/json' \
    -H 'ngsild-tenant: airqualityalert' \
    -H 'fiware-service: airqualityalert' \
    -d @sub.json

Notice the extra header of fiware-service: airqualityalert. The sub.json file:

{
    "id": "urn:ngsi-ld:Subscription:63dbab3998657c37179501a9",
    "type": "Subscription",
    "description": "Notify alert",
    "entities": [
      {
        "type": "AirQualityAlert"
      }
    ],
    "notification": {
      "attributes": [
        "analysis",
        "dateIssued",
        "prediction"
      ],
      "format": "normalized",
      "endpoint": {
        "uri": "http://quantumleap:8668/v2/notify",
        "accept": "application/json"
      }
    }
}

Now with the subscription in place, you can start sending your entities to Orion

$ curl 'http://localhost:1026/ngsi-ld/v1/entityOperations/upsert' \
    -H 'Content-Type: application/json' \
    -H 'ngsild-tenant: airqualityalert' \
    -H 'fiware-service: airqualityalert' \
    -H 'fiware-servicepath: /' \
    -d @data.json

Again, notice the extra header of fiware-service: airqualityalert. The data.json file:

[
    {
        "id": "urn:ngsi-ld:AirQualityAlert:Out.3857499894.1",
        "type": "AirQualityAlert",
        "analysis": {
            "type": "Property",
            "value": "good"
        },
        "prediction": {
            "type": "Property",
            "value": "good"
        },
        "dateIssued": {
            "type": "Property",
            "value": {
                "type": "DateTime",
                "value": "2023-02-07T15:57:20.519Z"
            }
        }
    }
]

Finally get back into Postgres to check QL stored your data

$ psql postgres://quantumleap:*@localhost

select * from mtairqualityalert.etairqualityalert;
                  entity_id                   |   entity_type   |          time_index           | fiware_servicepath | __original_ngsi_entity__ |                    instanceid                    | analysis | prediction |                        dateissued                         
----------------------------------------------+-----------------+-------------------------------+--------------------+--------------------------+--------------------------------------------------+----------+------------+-----------------------------------------------------------
 urn:ngsi-ld:AirQualityAlert:Out.3857499894.1 | AirQualityAlert | 2023-02-08 14:28:08.825083+00 | /                  |                          | urn:ngsi-ld:3a196511-cea1-4ab1-992e-398c9c62f41c | good     | good       | {"type": "DateTime", "value": "2023-02-07T15:57:20.519Z"}
 urn:ngsi-ld:AirQualityAlert:Out.3857499894.1 | AirQualityAlert | 2023-02-08 14:28:12.170136+00 | /                  |                          | urn:ngsi-ld:44de05ac-138c-4f77-af87-d04972fcc18f | good     | good       | {"type": "DateTime", "value": "2023-02-07T15:57:20.519Z"}
 urn:ngsi-ld:AirQualityAlert:Out.3857499894.1 | AirQualityAlert | 2023-02-08 14:28:15.281166+00 | /                  |                          | urn:ngsi-ld:ae6e6a91-27d0-40d2-b8aa-ae2db73154a3 | good     | good       | {"type": "DateTime", "value": "2023-02-07T15:57:20.519Z"}
(3 rows)

Hope this helps!

alvaroo278 commented 1 year ago

3. Add an additional fiware-service: airqualityalert header to your subscription and upsert Orion calls. This is b/c at the moment Quantum Leap still doesn't support the ngsild-tenant header. But this feature should be merged into the mainline soon, see [ngsild-tenant header support | Fix for issue #669 & #664 #695].

I discuss this with Ken Zangelin, and he said me that now NGSI-LD don't support header 'fiware-service': For backwards compatibility, I accept FIWARE-Service (and save it as NGSILD-Tenant) while the service-path is hardcoded to "/".

c0c0n3 commented 1 year ago

@alvaroo278 yea, I agree w/ you and Ken. The extra fiware-service header I suggested to put in is not a good idea in general. I was just suggesting a stop gap solution until we merge #695.

orchestracities / ngsi-timeseries-api

Problem to notify Quantunleap #719