search

FIWARE / context.Orion-LD

Context Broker and CEF building block for context data management which supports both the NGSI-LD and the NGSI-v2 APIs
https://www.etsi.org/deliver/etsi_gs/CIM/001_099/009/01.06.01_60/gs_CIM009v010601p.pdf
GNU Affero General Public License v3.0
50 stars 43 forks source link

Subscription at Orion-LD fails with Mongo-DB error #1160

Closed IngMiad closed 2 years ago

IngMiad commented 2 years ago

Hi there I already asked a question at stack overflow and have been asked to open an issue at Github. Therefore I open this issue related to question

I try to get a subscription from Orion-LD to Quantum Leap and CrateDB running. Unfortunately it seems that MongoDB throws an error Error (string field 'csf' is missing in BSONObj or Error (string field 'name' is missing in BSONObj' when it tries to access the subscription. The result is that the data can't be passed to Quantum Leap for further processing.

time=Friday 24 Jun 11:03:59 2022.081Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[145]:getStringField | msg=Runtime Error (string field 'name' is missing in BSONObj <{ _id: "urn:ngsi-ld:Subscription:62b578ea4567412cdf07306e", expiration: 2147483647.0, reference: "http://172.18.1.5:8668/v2/notify", custom: false, mimeType: "application/json", throttling: 0.0, servicePath: "/", description: "Notify me of temperature", status: "active", entities: [ { id: "", isPattern: "", type: "https://uri.fiware.org/ns/data-models#WeatherObserved", isTypePattern: false } ], attrs: [ "https://uri.fiware.org/ns/data-models#temperature" ], metadata: [], blacklist: false, ldContext: "http://172.18.1.2/datamodels.context-ngsi.jsonld", createdAt: 1656060138.058298, modifiedAt: 1656060138.058298, conditions: [ "https://uri.fiware.org/ns/data-models#temperature" ], expression: { q: "https://uri=fiware=org/ns/data-models#temperature<100", mq: "", geometry: "", coords: "", georel: "", geoproperty: "" }, format: "normalized" }> from caller setName:280)
time=Friday 24 Jun 11:03:59 2022.081Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[145]:getStringField | msg=Runtime Error (string field 'csf' is missing in BSONObj <{ _id: "urn:ngsi-ld:Subscription:62b578ea4567412cdf07306e", expiration: 2147483647.0, reference: "http://172.18.1.5:8668/v2/notify", custom: false, mimeType: "application/json", throttling: 0.0, servicePath: "/", description: "Notify me of temperature", status: "active", entities: [ { id: "", isPattern: "", type: "https://uri.fiware.org/ns/data-models#WeatherObserved", isTypePattern: false } ], attrs: [ "https://uri.fiware.org/ns/data-models#temperature" ], metadata: [], blacklist: false, ldContext: "http://172.18.1.2/datamodels.context-ngsi.jsonld", createdAt: 1656060138.058298, modifiedAt: 1656060138.058298, conditions: [ "https://uri.fiware.org/ns/data-models#temperature" ], expression: { q: "https://uri=fiware=org/ns/data-models#temperature<100", mq: "", geometry: "", coords: "", georel: "", geoproperty: "" }, format: "normalized" }> from caller setCsf:302)

Before that I created and validated the created subscription using

Creation: http://localhost:1026/ngsi-ld/v1/subscriptions/

{
  "description": "Notify me of temperature",
  "type": "Subscription",
  "entities": [{"type": "WeatherObserved"}],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://172.18.1.5:8668/v2/notify",
      "accept": "application/json"
    }
  },
   "@context": "http://172.18.1.2/datamodels.context-ngsi.jsonld"
}

Validation: http://localhost:1026/ngsi-ld/v1/subscriptions/

[
    {
        "id": "urn:ngsi-ld:Subscription:62b578ea4567412cdf07306e",
        "type": "Subscription",
        "description": "Notify me of temperature",
        "entities": [
            {
                "type": "WeatherObserved"
            }
        ],
        "watchedAttributes": [
            "temperature"
        ],
        "notification": {
            "attributes": [
                "temperature"
            ],
            "format": "normalized",
            "endpoint": {
                "uri": "http://172.18.1.5:8668/v2/notify",
                "accept": "application/json"
            }
        },
        "@context": "http://172.18.1.2/datamodels.context-ngsi.jsonld"
    },

So it seems that the subscription actually exists and that orion context broker tries to access it actively according to the MongoDB error in Orion-LD logs (docker logs -f ).

Could this be related to an old MongoDB driver? I found similar issues for older versions at https://github.com/telefonicaid/fiware-orion/issues/3070.

Here is my actual docker-compose file:

version: "3.8"
services:

  ########
  # CORE #
  ########
  # -> Orion: context broker as central component
  orion:
    labels:
      org.test: 'fiware'
    image: fiware/orion-ld:${ORION_VERSION}
    hostname: orion
    container_name: fiware-orion
    depends_on:
      - mongo-db
    networks:
      default:
        ipv4_address: 172.18.1.3
    ports:
      - "${ORION_PORT}:${ORION_PORT}"
    command: -dbhost mongo-db -logLevel DEBUG -noCache
    healthcheck:
      test: curl --fail -s http://orion:${ORION_PORT}/version || exit 1
      interval: 5s

  # -> Context: provide ngsi-ld context file for smart data models 
  ld-context:
    labels:
      org.test: 'fiware'
    image: httpd:alpine
    hostname: context
    container_name: fiware-ld-context
    ports:
        - "3004:80"
    networks:
      default:
        ipv4_address: 172.18.1.2
    volumes:
        - ./context:/usr/local/apache2/htdocs/
    healthcheck:
      test: (wget --server-response --spider --quiet  http://172.18.1.2/datamodels.context-ngsi.jsonld 2>&1 | awk 'NR==1{print $$2}'|  grep -q -e "200") || exit 1

  ##################
  # DATA MANGEMENT #
  ##################
  # Quantum Leap: is persisting Short Term History to Crate-DB
  quantumleap:
    labels:
      org.test: 'fiware'
    image: orchestracities/quantumleap:${QUANTUMLEAP_VERSION}
    hostname: quantumleap
    container_name: fiware-quantumleap
    depends_on:
      - crate-db
      - redis-db
    networks:
      default:
        ipv4_address: 172.18.1.5
    ports:
      - "${QUANTUMLEAP_PORT}:${QUANTUMLEAP_PORT}"
    environment:
      - CRATE_HOST=crate-db
      - REDIS_HOST=redis-db
      - REDIS_PORT=${REDIS_PORT}
      - LOGLEVEL=DEBUG
    healthcheck:
      test: curl --fail -s http://quantumleap:${QUANTUMLEAP_PORT}/version || exit 1

  #################
  # VISUALIZATION #
  #################
  # -> Grafana: Visualize Time Series data
  grafana:
    labels:
      org.test: 'fiware'
    image: grafana/grafana:6.1.6
    container_name: grafana
    depends_on:
      - crate-db
    networks:
      default:
        ipv4_address: 172.18.1.8
    ports:
      - "3003:3000"
    environment:
      - GF_INSTALL_PLUGINS=https://github.com/orchestracities/grafana-map-plugin/archive/master.zip;grafana-map-plugin,grafana-clock-panel,grafana-worldmap-panel
    volumes:
      - grafana:/var/lib/grafana

  #############
  # DATABASES #
  #############
  # -> MongoDB: database of Orion
  mongo-db:
    labels:
      org.test: 'fiware'
    image: mongo:${MONGO_DB_VERSION}
    hostname: mongo-db
    container_name: db-mongo
    expose:
      - "${MONGO_DB_PORT}"
    ports:
      - "${MONGO_DB_PORT}:${MONGO_DB_PORT}" # localhost:27017 # localhost:27017
    networks:
      default:
        ipv4_address: 172.18.1.4
    volumes:
      - mongo-db:/data
    healthcheck:
      test: |
        host=`hostname --ip-address || echo '127.0.0.1'`; 
        mongo --quiet $host/test --eval 'quit(db.runCommand({ ping: 1 }).ok ? 0 : 2)' && echo 0 || echo 1
      interval: 5s

  # -> CreateDB: database to store time-series data
  crate-db:
    labels:
      org.test: 'fiware'
    image: crate:${CRATE_VERSION}
    hostname: crate-db
    container_name: db-crate
    networks:
      default:
        ipv4_address: 172.18.1.6
    ports:
      # Admin UI
      - "4200:4200"
      # Transport protocol
      - "4300:4300"
    command: crate -Cauth.host_based.enabled=false  -Ccluster.name=democluster -Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
    environment:
      - CRATE_HEAP_SIZE=2g # see https://crate.io/docs/crate/howtos/en/latest/deployment/containers/docker.html#troubleshooting
    volumes:
      - crate-db:/data

  # -> Redis: Normally used to efficiently store key value pairs
  redis-db:
    labels:
      org.test: 'fiware'
    image: redis:${REDIS_VERSION}
    hostname: redis-db
    container_name: db-redis
    networks:
      default:
        ipv4_address: 172.18.1.7
    ports:
      - "${REDIS_PORT}:${REDIS_PORT}" # localhost:6379
    volumes:
      - redis-db:/data
    healthcheck:
      test: |
        host=`hostname -i || echo '127.0.0.1'`;
        ping=`redis-cli -h "$host" ping` && [ "$ping" = 'PONG' ] && echo 0 || echo 1
      interval: 10s

# NETWORKS
networks:
  default:
    labels:
      org.test: 'fiware'
    ipam:
      config:
        - subnet: 172.18.1.0/24

# VOLUMES
volumes:
  mongo-db: ~
  context: ~
  grafana: ~
  crate-db: ~
  redis-db: ~

And my related .env-file:

# Project name
COMPOSE_PROJECT_NAME=fiware

# Orion variables
ORION_PORT=1026
ORION_VERSION=1.0.0

# MongoDB variables
MONGO_DB_PORT=27017
MONGO_DB_VERSION=4.4

# QuantumLeap Variables
QUANTUMLEAP_VERSION=0.8.3
QUANTUMLEAP_PORT=8668

# CrateDB Version
CRATE_VERSION=4.6

# RedisDB Version
REDIS_PORT=6379
REDIS_VERSION=6

I would be really appreciated for further help as this issue already blocks me for some days. Thank you in advance!

kzangeli commented 2 years ago

ok, so, my first question would be: "does QL receive notifications"?

Because, this "csf" field isn't used for "normal" subscriptions - it is for subscribing to registrations, something that Orion-LD still doesn't implement. The ERROR you see in the log file if unfortunate, I will remove it asap. It's not an error, its the implementation trying to extract "csf" from the BSON object it got from mongo, and it is not there. Unfortunately the function used for extraction logs an error whenever the wanted field is not present. Ignore that (or add a "csf" field in the subscription and the "error" will go away). Same same for "name" - both are optional fields for Subscriptions.

So, if you create/update an entity matching the subscription, what happens?

kzangeli commented 2 years ago

I tried to remove the erroneous error but it seems I've done it already. I believe you're running a broker older than March 29 2022 (that's when git tells me I fixed this unfortunate false error).

So, shall we see if we can understand what's really happening ... ? Perhaps, pass me the entire logfile after:

  1. Creating the subscription
  2. Creating an entity that matches the subscription snd should provoke a notification to QL (well, or wherever).

And, of course, complete info on the subscription created (entire payload body and HTTP headers) and the Entity to provoke the notification.

Give me that and I'll look into it

Also, I think I saw some Q-filter in the Stackoverflow issue ("q": "https://uri.fiware.org/ns/data-models#temperature<100"). Make sure you give me exact information - there is no way you can create a subscription without 'q' and then GET it with a perfectly valid 'q' ... no way. That 'q' is a filter that could be what's preventing notification to be sent ...

IngMiad commented 2 years ago

Hello kzangeli,

thank you for the information. It seems that I messed up my example a bit, because I tried several subscription (with and without q). But both are not working in my case and QL doesn't receive any notification. I can also not find any error in the logs other than the csf / name error...

I tried a fresh minimal example with the official Weather Smart Data Model and related example with the docker-compose file from my first request (to make sure that my data model is not the problem):

  1. Create a subscription at Orion for QL

       curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: test' \
--data-raw '{
  "description": "Notify me of temperature by example (without q)",
  "type": "Subscription",
  "entities": [{"type": "WeatherObserved"}],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://172.18.1.5:8668/v2/notify",
      "accept": "application/json"
    }
  },
   "@context": "https://smart-data-models.github.io/dataModel.Weather/context.jsonld"
}'

  2. Check created subscription a. Request

    curl --location --request GET 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
--header 'NGSILD-Tenant: test'

    b. Response

    [
    {
        "id": "urn:ngsi-ld:Subscription:62bd6a6c80620e80336e6384",
        "type": "Subscription",
        "description": "Notify me of temperature by example (without q)",
        "entities": [
            {
                "type": "WeatherObserved"
            }
        ],
        "watchedAttributes": [
            "temperature"
        ],
        "notification": {
            "attributes": [
                "temperature"
            ],
            "format": "normalized",
            "endpoint": {
                "uri": "http://172.18.1.5:8668/v2/notify",
                "accept": "application/json"
            }
        },
        "@context": "https://smart-data-models.github.io/dataModel.Weather/context.jsonld"
    }
]

  3. Create a Weather entity (example from: https://smart-data-models.github.io/dataModel.Weather/WeatherObserved/examples/example-normalized.jsonld)

    curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: test' \
--data-raw '{
  "id": "urn:ngsi-ld:WeatherObserved:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:06.00Z",
  "type": "WeatherObserved",
  "address": {
    "type": "Property",
    "value": {
      "addressLocality": "Valladolid",
      "addressCountry": "ES",
      "type": "PostalAddress"
    }
  },
  "atmosphericPressure": {
    "type": "Property",
    "value": 938.9
  },
  "dataProvider": {
    "type": "Property",
    "value": "TEF"
  },
  "dateObserved": {
    "type": "Property",
    "value": {
      "@type": "DateTime",
      "@value": "2016-11-30T07:00:00.00Z"
    }
  },
  "illuminance": {
    "type": "Property",
    "value": 1000
  },
  "location": {
    "type": "GeoProperty",
    "value": {
      "type": "Point",
      "coordinates": [
        -4.754444444,
        41.640833333
      ]
    }
  },
  "precipitation": {
    "type": "Property",
    "value": 0
  },
  "pressureTendency": {
    "type": "Property",
    "value": 0.5
  },
  "refDevice": {
    "type": "Relationship",
    "object": "urn:ngsi-ld:Device:device-0A3478"
  },
  "relativeHumidity": {
    "type": "Property",
    "value": 1
  },
  "snowHeight": {
    "type": "Property",
    "value": 20
  },
  "source": {
    "type": "Property",
    "value": "http://www.aemet.es"
  },
  "stationCode": {
    "type": "Property",
    "value": "2422"
  },
  "stationName": {
    "type": "Property",
    "value": "Valladolid"
  },
  "streamGauge": {
    "type": "Property",
    "value": 50
  },
  "temperature": {
    "type": "Property",
    "value": 3.3
  },
  "uvIndexMax": {
    "type": "Property",
    "value": 1.0
  },
  "windDirection": {
    "type": "Property",
    "value": 135
  },
  "windSpeed": {
    "type": "Property",
    "value": 2
  },
  "@context": [
    "https://smart-data-models.github.io/dataModel.Weather/context.jsonld"
  ]
}'

  4. Check if entity has been created successfully a. Request

    curl --location --request GET 'http://localhost:1026/ngsi-ld/v1/entities/?type=WeatherObserved&options=keyValues' \
--header 'Link: <https://smart-data-models.github.io/dataModel.Weather/context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--header 'NGSILD-Tenant: test'

    b. Response

    [
    {
        "@context": "https://smart-data-models.github.io/dataModel.Weather/context.jsonld",
        "id": "urn:ngsi-ld:WeatherObserved:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:06.00Z",
        "type": "WeatherObserved",
        "address": {
            "addressLocality": "Valladolid",
            "addressCountry": "ES",
            "type": "PostalAddress"
        },
        "atmosphericPressure": 938.9,
        "illuminance": 1000,
        "precipitation": 0,
        "pressureTendency": 0.5,
        "refDevice": "urn:ngsi-ld:Device:device-0A3478",
        "relativeHumidity": 1,
        "snowHeight": 20,
        "streamGauge": 50,
        "temperature": 3.3,
        "windDirection": 135,
        "windSpeed": 2,
        "dataProvider": "TEF",
        "dateObserved": {
            "@type": "DateTime",
            "@value": "2016-11-30T07:00:00.00Z"
        },
        "source": "http://www.aemet.es",
        "stationCode": "2422",
        "stationName": "Valladolid",
        "uvIndexMax": 1,
        "location": {
            "type": "Point",
            "coordinates": [
                -4.754444,
                41.640833
            ]
        }
    }
]

  5. Check if anything exists at QL a. Request

    curl --location --request GET 'http://localhost:8668/v2/entities/urn:ngsi-ld:WeatherObserved:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:06.00Z/attrs/temperature' \
--header 'Accept: application/json' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /'

    b. Response

    {
"description": "No records were found for such query.",
"error": "Not Found"
}

Again no errors when checking orion / QL with docker logs -f <container>. I am not sure what is going wrong and where I should continue to debug. Tell me if you need any additional information. Thank you in advance!

kzangeli commented 2 years ago

ok. I believe your info is not 100% complete when it comes to the @context used in every request. Remember that the attribute names are expanded using the @context.

That means that if you create a subscription with

  "watchedAttributes": "temperature"

And an @context that expands "temperature" to "http://a.b.c/attrs/temp", and then you create an entity with a "matching" attribute "temperature" but with a different @context, expanding "temperature" to "http://d.e.f/attrs/temp", then you will get no notification, as those two "temperature" attributes are actually two different attributes. Same same with entity types - the value of the entity type is expanded the same way as attribute names.

Also, be careful with the tenants - thoser are different, isolated, databases, and an entity in tenant A will never notify on a subscription from tenant B.

If all that is "under control", then I would ask you to send me the traces of your broker and I'll see if I can find anything.

IngMiad commented 2 years ago

Hello again,

okay I checked on both things and I think it is better to go back to example one with my own validated and created data model rather than the https://smart-data-models.github.io/dataModel.Weather/context.jsonld in the second example. Sorry for the mess.

The static alpine context service provides following ngsi-ld file in my example at http://172.18.1.2/datamodels.context-ngsi.jsonld:

{
    "@context": {
        "type": "@type",
        "id": "@id",
        "ngsi-ld": "https://uri.etsi.org/ngsi-ld/",
        "fiware": "https://uri.fiware.org/ns/data-models#",
        "schema": "https://schema.org/",
        "WeatherObserved": "fiware:WeatherObserved",
        "address": "fiware:address",
        "airQualityIndex": "fiware:airQualityIndex",
        "airQualityIndexForecast": "fiware:airQualityIndexForecast",
        "airTemperatureForecast": "fiware:airTemperatureForecast",
        "airTemperatureTSA": "fiware:airTemperatureTSA",
        "alternateName": "fiware:alternateName",
        "aqiMajorPollutant": "fiware:aqiMajorPollutant",
        "aqiMajorPollutantForecast": "fiware:aqiMajorPollutantForecast",
        "areaServed": "fiware:areaServed",
        "atmosphericPressure": "fiware:atmosphericPressure",
        "dataProvider": "fiware:dataProvider",
        "dateCreated": "fiware:dateCreated",
        "dateModified": "fiware:dateModified",
        "dateObserved": "fiware:dateObserved",
        "description": "fiware:description",
        "dewPoint": "fiware:dewPoint",
        "feelLikesTemperature": "fiware:feelLikesTemperature",
        "gustSpeed": "fiware:gustSpeed",
        "illuminance": "fiware:illuminance",
        "name": "fiware:name",
        "owner": "fiware:owner",
        "precipitation": "fiware:precipitation",
        "precipitationForecast": "fiware:precipitationForecast",
        "pressureTendency": "fiware:pressureTendency",
        "refDevice": "fiware:refDevice",
        "refPointOfInterest": "fiware:refPointOfInterest",
        "relativeHumidity": "fiware:relativeHumidity",
        "relativeHumidityForecast": "fiware:relativeHumidityForecast",
        "seeAlso": "fiware:seeAlso",
        "snowHeight": "fiware:snowHeight",
        "solarRadiation": "fiware:solarRadiation",
        "source": "fiware:source",
        "streamGauge": "fiware:streamGauge",
        "temperature": "fiware:temperature",
        "uVIndexMax": "fiware:uVIndexMax",
        "visibility": "fiware:visibility",
        "weatherType": "fiware:weatherType",
        "windDirection": "fiware:windDirection",
        "windSpeed": "fiware:windSpeed"
    }
}

It's a simple model just referencing all attributes of WeatherObserved Entity at Smart Data Model Weather. I validated it successfully using smart data model tool and then created the ngsi-ld file you see above.

To keep it short we actually need only 3 queries to fire subscriptions for the scenario: 1) Create Subscription

curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: test' \
--data-raw '{
  "description": "Notify me of temperature by example (without q)",
  "type": "Subscription",
  "entities": [{"type": "WeatherObserved"}],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://172.18.1.5:8668/v2/notify",
      "accept": "application/json"
    }
  },
   "@context": "http://172.18.1.2/datamodels.context-ngsi.jsonld"
}'

2) Create Entity

curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/ld+json' \
--header 'NGSILD-Tenant: test' \
--data-raw '{
  "id": "urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z",
  "type": "WeatherObserved",
  "address": {
    "type": "Property",
    "value": {
      "addressLocality": "Valladolid",
      "addressCountry": "ES",
      "type": "PostalAddress"
    }
  },
  "atmosphericPressure": {
    "type": "Property",
    "value": 938.9
  },
  "dataProvider": {
    "type": "Property",
    "value": "TEF"
  },
  "dateObserved": {
    "type": "Property",
    "value": {
      "@type": "DateTime",
      "@value": "2016-11-30T07:00:00.00Z"
    }
  },
  "illuminance": {
    "type": "Property",
    "value": 1000
  },
  "location": {
    "type": "GeoProperty",
    "value": {
      "type": "Point",
      "coordinates": [
        -4.754444444,
        41.640833333
      ]
    }
  },
  "precipitation": {
    "type": "Property",
    "value": 0
  },
  "pressureTendency": {
    "type": "Property",
    "value": 0.5
  },
  "refDevice": {
    "type": "Relationship",
    "object": "urn:ngsi-ld:Device:device-0A3478"
  },
  "relativeHumidity": {
    "type": "Property",
    "value": 1
  },
  "snowHeight": {
    "type": "Property",
    "value": 20
  },
  "source": {
    "type": "Property",
    "value": "http://www.aemet.es"
  },
  "stationCode": {
    "type": "Property",
    "value": "2422"
  },
  "stationName": {
    "type": "Property",
    "value": "Valladolid"
  },
  "streamGauge": {
    "type": "Property",
    "value": 50
  },
  "temperature": {
    "type": "Property",
    "value": 10.3
  },
  "uvIndexMax": {
    "type": "Property",
    "value": 1.0
  },
  "windDirection": {
    "type": "Property",
    "value": 135
  },
  "windSpeed": {
    "type": "Property",
    "value": 3
  },
  "@context": [
    "http://172.18.1.2/datamodels.context-ngsi.jsonld"
  ]
}'

3) Check Entity at Quantum Leap

curl --location --request GET 'http://localhost:8668/v2/entities/urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z/attrs/temperature' \
--header 'Accept: application/json' \
--header 'fiware-service: test' \
--header 'fiware-servicepath: /'

The NGSILD-Tenant / fiware-service 'test' is the same in all requests and the related @context file looks okay (as far I can judge on that).

I am not sue if that might help: When I check docker logs -f fiware-orion I can see the long names for the fields from mongo-db error when it tries to access the subscription: 

time=Friday 01 Jul 14:21:04 2022.704Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[145]:getStringField | msg=Runtime Error (string field 'name' is missing in BSONObj <{ _id: "urn:ngsi-ld:Subscription:62bf02bf7631ee58c4f7095d", expiration: 2147483647.0, reference: "http://172.18.1.5:8668/v2/notify", custom: false, mimeType: "application/json", throttling: 0.0, servicePath: "/", description: "Notify me of temperature by example (without q)", status: "active", entities: [ { id: "", isPattern: "", type: "https://uri.fiware.org/ns/data-models#WeatherObserved", isTypePattern: false } ], attrs: [ "https://uri.fiware.org/ns/data-models#temperature" ], metadata: [], blacklist: false, ldContext: "http://172.18.1.2/datamodels.context-ngsi.jsonld", createdAt: 1656685247.597414, modifiedAt: 1656685247.597414, conditions: [ "https://uri.fiware.org/ns/data-models#temperature" ], expression: { q: "", mq: "", geometry: "", coords: "", georel: "", geoproperty: "" }, format: "normalized" }> from caller setName:280)

It seems to resolve the entity / attribute as https://uri.fiware.org/ns/data-models#WeatherObservedand https://uri.fiware.org/ns/data-models#temperature.

The subscription is still not being triggered and nothing arrives at QL. I am thankful for all suggestions for further analysis.

Greetings Inga

kzangeli commented 2 years ago

So, the subscription and entity look good to me, should fire a notification. One doubt: you say that nothing arrives at QL Do you base that on the query you do to QL? Or, do you base it on traces from Orion-LD and/or QL?

Cause the query to QL seems to be done via NGSIv2, and the attribute name "temperature". The name of the attribute is "https://uri.fiware.org/ns/data-models#temperature", and nothing else. NGSIv2 knows nothing about expansion, so that might be why. Does QL support "GET http://localhost:8668/v2/entities/urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z" ?

I will copy your example, just replacing your @context with "no context at all, i.e. the Core Context", and QL with a simply python script I use for tests. I will let you know shortly - time to make this work!

kzangeli commented 2 years ago

Functest implemented an all good - notification arrives for my notification-receiving test script. I also didn't use any tenants. Stripped it down to the minimum, which is what I'd propose you to do.

Just to try to make it work:

Then once that works, introduce "complications", namely tenant and your own user @context. Wherever it stops working, that's your problem right there.

My functest looks like this, hopefully quite self-explanatory:

--SHELL-INIT--
export BROKER=orionld
dbInit CB
brokerStart CB
accumulatorStart --pretty-print 127.0.0.1 ${LISTENER_PORT}

--SHELL--

#
# 01. Create a subscription for the accumulator 
# 02. Create a matching entity
# 03. Dump the accumulator to see the notification
#

echo "01. Create a subscription for the accumulator"
echo "============================================="
payload='{
  "description": "Notify me of temperature by example (without q)",
  "type": "Subscription",
  "entities": [{"type": "WeatherObserved"}],
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "format": "normalized",
    "endpoint": {
      "uri": "http://127.0.0.1:'${LISTENER_PORT}'/notify",
      "accept": "application/json"
    }
  }
}'
orionCurl --url /ngsi-ld/v1/subscriptions --payload "$payload"
echo
echo

echo "02. Create a matching entity"
echo "============================"
payload='{
  "id": "urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z",
  "type": "WeatherObserved",
  "address": {
    "type": "Property",
    "value": {
      "addressLocality": "Valladolid",
      "addressCountry": "ES",
      "type": "PostalAddress"
    }
  },
  "atmosphericPressure": {
    "type": "Property",
    "value": 938.9
  },
  "dataProvider": {
    "type": "Property",
    "value": "TEF"
  },
  "dateObserved": {
    "type": "Property",
    "value": {
      "@type": "DateTime",
      "@value": "2016-11-30T07:00:00.00Z"
    }
  },
  "illuminance": {
    "type": "Property",
    "value": 1000
  },
  "location": {
    "type": "GeoProperty",
    "value": {
      "type": "Point",
      "coordinates": [
        -4.754444444,
        41.640833333
      ]
    }
  },
  "precipitation": {
    "type": "Property",
    "value": 0
  },
  "pressureTendency": {
    "type": "Property",
    "value": 0.5
  },
  "refDevice": {
    "type": "Relationship",
    "object": "urn:ngsi-ld:Device:device-0A3478"
  },
  "relativeHumidity": {
    "type": "Property",
    "value": 1
  },
  "snowHeight": {
    "type": "Property",
    "value": 20
  },
  "source": {
    "type": "Property",
    "value": "http://www.aemet.es"
  },
  "stationCode": {
    "type": "Property",
    "value": "2422"
  },
  "stationName": {
    "type": "Property",
    "value": "Valladolid"
  },
  "streamGauge": {
    "type": "Property",
    "value": 50
  },
  "temperature": {
    "type": "Property",
    "value": 10.3
  },
  "uvIndexMax": {
    "type": "Property",
    "value": 1.0
  },
  "windDirection": {
    "type": "Property",
    "value": 135
  },
  "windSpeed": {
    "type": "Property",
    "value": 3
  }
}'
orionCurl --url /ngsi-ld/v1/entities --payload "$payload"
echo
echo

echo "03. Dump the accumulator to see the notification"
echo "================================================"
accumulatorDump
echo
echo

--REGEXPECT--
01. Create a subscription for the accumulator
=============================================
HTTP/1.1 201 Created
Content-Length: 0
Date: REGEX(.*)
Location: /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:REGEX(.*)

02. Create a matching entity
============================
HTTP/1.1 201 Created
Content-Length: 0
Date: REGEX(.*)
Location: /ngsi-ld/v1/entities/urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z

03. Dump the accumulator to see the notification
================================================
POST REGEX(.*)
Fiware-Servicepath: /
Content-Length: 380
User-Agent: orion/REGEX(.*)
Ngsiv2-Attrsformat: normalized
Host: 127.0.0.1:9997
Accept: application/json
Content-Type: application/json; charset=utf-8
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"

{
    "data": [
        {
            "id": "urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z",
            "temperature": {
                "type": "Property",
                "value": 10.3
            },
            "type": "WeatherObserved"
        }
    ],
    "id": "urn:ngsi-ld:Notification:REGEX(.*)",
    "notifiedAt": "202REGEX(.*)",
    "subscriptionId": "urn:ngsi-ld:Subscription:REGEX(.*)",
    "type": "Notification"
}
=======================================

--TEARDOWN--
brokerStop CB
accumulatorStop
dbDrop CB
IngMiad commented 2 years ago

Hello,

Just to try to make it work: - No Context - just the core context that is built'in inside the broker - No Tenants - jsut the default on, also "built-in" inside the broker - Find a better way to find out whether QL received the notification of not - querying it about an attribute is not good enough.

Then once that works, introduce "complications", namely tenant and your own user @context. Wherever it stops working, that's your problem right there."

okay sounds good to me.

I haven't known about accumulator so far, but that might be useful to check the subscription / notification. Therefore I take an accumulator file https://raw.githubusercontent.com/telefonicaid/fiware-orion/master/scripts/accumulator-server.py (just called accumulator in my case) and started the server as you did: 

./accumulator accumulatorStart --pretty-print 127.0.0.1 1028
 * Serving Flask app 'accumulator' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses (0.0.0.0)
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://127.0.0.1:1028
 * Running on http://192.168.1.200:1028 (Press CTRL+C to quit)

Now the three requests:

  1. Create subscription without context and tenant 
    curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
--header 'Content-Type: application/json' \
--data-raw '{
"description": "Notify me of temperature by example (without q)",
"type": "Subscription",
"entities": [{"type": "WeatherObserved"}],
"watchedAttributes": ["temperature"],
"notification": {
"attributes": ["temperature"],
"format": "normalized",
"endpoint": {
  "uri": "http://127.0.0.1:1028/notify",
  "accept": "application/json"
}
}
}'
  2. Create an entity without context and tenant 
    curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z",
"type": "WeatherObserved",
"address": {
"type": "Property",
"value": {
  "addressLocality": "Valladolid",
  "addressCountry": "ES",
  "type": "PostalAddress"
}
},
"atmosphericPressure": {
"type": "Property",
"value": 938.9
},
"dataProvider": {
"type": "Property",
"value": "TEF"
},
"dateObserved": {
"type": "Property",
"value": {
  "@type": "DateTime",
  "@value": "2016-11-30T07:00:00.00Z"
}
},
"illuminance": {
"type": "Property",
"value": 1000
},
"location": {
"type": "GeoProperty",
"value": {
  "type": "Point",
  "coordinates": [
    -4.754444444,
    41.640833333
  ]
}
},
"precipitation": {
"type": "Property",
"value": 0
},
"pressureTendency": {
"type": "Property",
"value": 0.5
},
"refDevice": {
"type": "Relationship",
"object": "urn:ngsi-ld:Device:device-0A3478"
},
"relativeHumidity": {
"type": "Property",
"value": 1
},
"snowHeight": {
"type": "Property",
"value": 20
},
"source": {
"type": "Property",
"value": "http://www.aemet.es"
},
"stationCode": {
"type": "Property",
"value": "2422"
},
"stationName": {
"type": "Property",
"value": "Valladolid"
},
"streamGauge": {
"type": "Property",
"value": 50
},
"temperature": {
"type": "Property",
"value": 10.3
},
"uvIndexMax": {
"type": "Property",
"value": 1.0
},
"windDirection": {
"type": "Property",
"value": 135
},
"windSpeed": {
"type": "Property",
"value": 3
}
}'
  3. Check accumulator dump 
    curl --location --request GET 'http://127.0.0.1:1028/dump'

Here are the responses:

  1. Create subscription for accumulator

    HTTP/1.1 201 Created
Connection: Keep-Alive
Content-Length: 0
Location: /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:62c2e4a5cde2ab0288d5df5e
Date: Mon, 04 Jul 2022 13:01:25 GMT

  2. Create Matching Entity

    HTTP/1.1 201 Created
Connection: Keep-Alive
Content-Length: 0
Location: /ngsi-ld/v1/entities/urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z
Date: Mon, 04 Jul 2022 13:03:11 GMT

  3. Response when I dump accumulator:

    HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.10.4
Date: Mon, 04 Jul 2022 13:03:49 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: close

Step 3 unfortunately returns 200 OK status code with an empty body.

I am not sure if am using this accumulator correctly. Executing accumulatorDump with cmd results in another starting accumulator with the script I use (it just starts the accumulator loop in all cases). Maybe I am not using the correct one or using it in a wrong way? I found the /dump url way of doing it in the header of the script:

# This program stores everything it receives by HTTP in a given URL (pased as argument),
# Then return the accumulated data upon receiving 'GET <host>:<port>/dump'. It is aimet
# at harness test for subscription scenarios (so accumulator-server.py plays the role
# of a subscribed application)
kzangeli commented 2 years ago

I think you have to start the accumulator with a parameter telling it the API endpoint "/notify".

Something like this:

scripts/accumulator-server.py --port 1028 --url "/notify" --host localhost
IngMiad commented 2 years ago

I already tried kind of with /accumulate as URL as explained here: https://fiware-orion.readthedocs.io/en/1.7.0/user/walkthrough_apiv1/#starting-accumulator-server-for-the-tutorials

When I try yor suggesting to startup accumulator in a different way, there is still nothing at accumulator:

  1. Start accumulator 
    troubleshooting/accumulator --port 1028 --url "/notify" --host localhost
* Serving Flask app 'accumulator' (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://localhost:1028 (Press CTRL+C to quit)
  2. Create Subscription 
    
REQUEST:
POST /ngsi-ld/v1/subscriptions/ HTTP/1.1
Content-Type: application/json
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 5f1eb302-2877-4de6-a1de-ac637df0cf3a
Host: localhost:1026
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 373

{ "description": "Notify me of temperature by example (without q)", "type": "Subscription", "entities": [{"type": "WeatherObserved"}], "watchedAttributes": ["temperature"], "notification": { "attributes": ["temperature"], "format": "normalized", "endpoint": { "uri": "http://127.0.0.1:1028/notify", "accept": "application/json" } } }

RESPONSE: HTTP/1.1 201 Created Connection: Keep-Alive Content-Length: 0 Location: /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:62c2ef7247b24bd5278767ec Date: Mon, 04 Jul 2022 13:47:30 GMT


3. Create Entity

REQUEST: POST /ngsi-ld/v1/entities/ HTTP/1.1 Content-Type: application/json User-Agent: PostmanRuntime/7.29.0 Accept: / Postman-Token: 00201223-43d0-4e5b-8eed-dc383ad0999d Host: localhost:1026 Accept-Encoding: gzip, deflate, br Connection: keep-alive Content-Length: 1704

{ "id": "urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z", "type": "WeatherObserved", "address": { "type": "Property", "value": { "addressLocality": "Valladolid", "addressCountry": "ES", "type": "PostalAddress" } }, "atmosphericPressure": { "type": "Property", "value": 938.9 }, "dataProvider": { "type": "Property", "value": "TEF" }, "dateObserved": { "type": "Property", "value": { "@type": "DateTime", "@value": "2016-11-30T07:00:00.00Z" } }, "illuminance": { "type": "Property", "value": 1000 }, "location": { "type": "GeoProperty", "value": { "type": "Point", "coordinates": [ -4.754444444, 41.640833333 ] } }, "precipitation": { "type": "Property", "value": 0 }, "pressureTendency": { "type": "Property", "value": 0.5 }, "refDevice": { "type": "Relationship", "object": "urn:ngsi-ld:Device:device-0A3478" }, "relativeHumidity": { "type": "Property", "value": 1 }, "snowHeight": { "type": "Property", "value": 20 }, "source": { "type": "Property", "value": "http://www.aemet.es" }, "stationCode": { "type": "Property", "value": "2422" }, "stationName": { "type": "Property", "value": "Valladolid" }, "streamGauge": { "type": "Property", "value": 50 }, "temperature": { "type": "Property", "value": 10.5 }, "uvIndexMax": { "type": "Property", "value": 1.0 }, "windDirection": { "type": "Property", "value": 135 }, "windSpeed": { "type": "Property", "value": 3 } }

RESPONSE: HTTP/1.1 201 Created Connection: Keep-Alive Content-Length: 0 Location: /ngsi-ld/v1/entities/urn:ngsi-ld:Weather:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:00.00Z Date: Mon, 04 Jul 2022 13:48:30 GMT


4. Check accumulator

REQUEST: GET http://127.0.0.1:1028/dump 200 25 ms GET /dump HTTP/1.1 User-Agent: PostmanRuntime/7.29.0 Accept: / Postman-Token: 3063f37a-c7c0-40ed-a0b0-6a795396de3c Host: 127.0.0.1:1028 Accept-Encoding: gzip, deflate, br Connection: keep-alive

RESPONSE: HTTP/1.1 200 OK Server: Werkzeug/2.1.2 Python/3.10.4 Date: Mon, 04 Jul 2022 13:49:13 GMT Content-Type: text/html; charset=utf-8 Content-Length: 0 Connection: close

kzangeli commented 2 years ago

That is quite strange. Let's look at the traces of the broker. Do you have anything interesting there?

IngMiad commented 2 years ago

Hello,

unfortunately I can not see any other error. If I execute docker logs -f fiware-orion I can only see the mongo-db error which will be shown when I create a subscription. Are there other error traces I could check?

Don't know if that might be interesting, but adding accumulator to my docker-compose and run it also as container changed the response in step 4 (check dump at accumulator) from empty to some http header information (still no subscription information unfortunately). Here is the extension to docker-compose:

  #########
  # DEBUG #
  #########
  accumulator:
    labels:
      org.test: 'fiware'
    build: ./troubleshooting
    hostname: accumulator
    container_name: accumulator
    networks:
      default:
        ipv4_address: 172.18.1.9
    ports:
        - "1028:1028"

I build it using following Dockerfile:

FROM python:3.8-slim-buster

# Set environment varibles
ENV PYTHONUNBUFFERED 1

# Set work directory
RUN mkdir /code
WORKDIR /code

COPY entrypoint.sh .
COPY accumulator .

# Install dependencies
RUN pip install --upgrade pip
RUN pip install flask
RUN pip install paho-mqtt

RUN chmod +x ./entrypoint.sh
RUN chmod +x ./accumulator

ENTRYPOINT ["./entrypoint.sh"]

And provide this entrypoint.sh:

#!/bin/bash

echo "Start accumulator"
./accumulator --port 1028 --url "/notify" --host 172.18.1.9 &>> accumulator.log

Afterwards I can use 172.18.1.9 to address accumulator container and do the same from last comment (create subscription, entity and then dump the accumulator -> just change localhost to ip address). This time the dump has still no information regarding the subscription, but it's body is at least filled with some Header information in comparison to last time (not as many as in your test, but some of them):

REQUEST:
GET /dump HTTP/1.1
User-Agent: PostmanRuntime/7.29.0
Accept: */*
Postman-Token: 3b46a83c-fcf3-4a60-b8ef-7f1f6f0bd8ef
Host: 172.18.1.9:1028
Accept-Encoding: gzip, deflate, br
Connection: keep-alive

RESPONSE:
HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.8.13
Date: Mon, 04 Jul 2022 15:42:07 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 252
Connection: close

GET http://172.18.1.9:1028/notify
User-Agent: PostmanRuntime/7.29.0
Host: 172.18.1.9:1028
Accept: */*
Postman-Token: 4ba4d248-c9f1-4bea-9790-5e1136a3e848
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
=======================================
IngMiad commented 2 years ago

As an addendum to my last comment I attach the logs from orion, but you can just see the mongo-db error when creating the subscription:

docker logs -f fiware-orion 
time=Wednesday 06 Jul 05:39:10 2022.074Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[925]:main | msg=Orion Context Broker is running
time=Wednesday 06 Jul 05:39:10 2022.074Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[756]:versionInfo | msg=Version Info:
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[757]:versionInfo | msg=-----------------------------------------
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[758]:versionInfo | msg=orionld version:    1.0.0
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[759]:versionInfo | msg=based on orion:     1.15.0-next
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[760]:versionInfo | msg=git hash:           nogitversion
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[761]:versionInfo | msg=build branch:       
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[762]:versionInfo | msg=compiled by:        root
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[763]:versionInfo | msg=compiled in:        
time=Wednesday 06 Jul 05:39:10 2022.075Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[764]:versionInfo | msg=-----------------------------------------
time=Wednesday 06 Jul 05:39:10 2022.134Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=MongoGlobal.cpp[245]:mongoInit | msg=Connected to mongo at mongo-db:orion
time=Wednesday 06 Jul 05:39:10 2022.573Z | lvl=INFO | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1075]:main | msg=Startup completed
time=Wednesday 06 Jul 05:39:10 2022.573Z | lvl=TMP | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=orionld.cpp[1083]:main | msg=Initialization ready - accepting REST requests on port 1026
time=Wednesday 06 Jul 05:42:29 2022.100Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[145]:getStringField | msg=Runtime Error (string field 'name' is missing in BSONObj <{ _id: "urn:ngsi-ld:Subscription:62c52051a2bce3a6d5542039", expiration: 2147483647.0, reference: "http://172.18.1.9:1028/notify", custom: false, mimeType: "application/json", throttling: 0.0, servicePath: "/", description: "Notify me of temperature at accumulator", status: "active", entities: [ { id: "", isPattern: "", type: "https://uri.etsi.org/ngsi-ld/default-context/WeatherObserved", isTypePattern: false } ], attrs: [ "https://uri.etsi.org/ngsi-ld/default-context/temperature" ], metadata: [], blacklist: false, ldContext: "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld", createdAt: 1657086033.959555, modifiedAt: 1657086033.959555, conditions: [ "https://uri.etsi.org/ngsi-ld/default-context/temperature" ], expression: { q: "", mq: "", geometry: "", coords: "", georel: "", geoproperty: "" }, format: "normalized" }> from caller setName:280)
time=Wednesday 06 Jul 05:42:29 2022.100Z | lvl=ERROR | corr=N/A | trans=N/A | from=N/A | srv=N/A | subsrv=N/A | comp=Orion | op=safeMongo.cpp[145]:getStringField | msg=Runtime Error (string field 'csf' is missing in BSONObj <{ _id: "urn:ngsi-ld:Subscription:62c52051a2bce3a6d5542039", expiration: 2147483647.0, reference: "http://172.18.1.9:1028/notify", custom: false, mimeType: "application/json", throttling: 0.0, servicePath: "/", description: "Notify me of temperature at accumulator", status: "active", entities: [ { id: "", isPattern: "", type: "https://uri.etsi.org/ngsi-ld/default-context/WeatherObserved", isTypePattern: false } ], attrs: [ "https://uri.etsi.org/ngsi-ld/default-context/temperature" ], metadata: [], blacklist: false, ldContext: "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context.jsonld", createdAt: 1657086033.959555, modifiedAt: 1657086033.959555, conditions: [ "https://uri.etsi.org/ngsi-ld/default-context/temperature" ], expression: { q: "", mq: "", geometry: "", coords: "", georel: "", geoproperty: "" }, format: "normalized" }> from caller setCsf:302)

I am using:

Maybe that makes a difference to the test environment where you tested it successfully? I could try to create same environment if this makes the difference. I have no further idea what I could try next to make this running.

kzangeli commented 2 years ago

Your versions seem fine. I think we need to have a session. Don't know how to help you any other way. Tomorrow morning I'm free (9-12) - July 7 Look me up on skype - the handle is kzangeli (you'll find two - ping me on both)

IngMiad commented 2 years ago

Okay sure, thank you in advance for your time. I also have time tomorrow morning and will contact you like you suggested.

wistefan commented 2 years ago

Hi @IngMiad , It looks as if you stumbled upon a couple of things;) I changed some parts to get it working. I guess you took the compose from one of the tutorials? Can you please send me a link on which one, so that we can update them properly?

To get it working:

  1. I removed crate-db as backend for quantum-leap. Crate-db has a number of issues and didn't even start at my machine. Its currently getting deprecated anyways, so that using time-scale is more appropriate. I use the following compose(see inline comments for explenation, I added all env vars directly for better readability):

version: "3.8" services:

########

CORE

########

-> Orion: context broker as central component

orion: labels: org.test: 'fiware' image: fiware/orion-ld:1.0.1 hostname: orion container_name: fiware-orion depends_on:

NETWORKS

networks: default: labels: org.test: 'fiware' ipam: config:

VOLUMES

volumes: mongo-db: ~ context: ~ grafana: ~ crate-db: ~ redis-db: ~ 


2. The following config.yaml needs to be available beside the docker-compose.yaml:
```yaml 
 default-backend: "timescale"
  1. I use the default context and no tenants in the curl, so that the whole setup becomes less complicated. They can be add back in the next step. The subscription is used with the sdn-name instead of the ip, since orion lives in the same network as quantum-leap. 
    curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/subscriptions' \
--header 'Content-Type: application/json' \
--data-raw '{
"description": "Notify me of temperature by example (without q)",
"type": "Subscription",
"entities": [{"type": "WeatherObserved"}],
"notification": {
 "attributes": ["temperature"],
 "format": "normalized",
 "endpoint": {
   "uri": "http://quantumleap:8668/v2/notify",
   "accept": "application/json"
 }
}
}'
  2. Create entity(same default context and no tenant): 
    curl --location --request POST 'http://localhost:1026/ngsi-ld/v1/entities/' \
--header 'Content-Type: application/json' \
--data-raw '{
"id": "urn:ngsi-ld:WeatherObserved:Spain-WeatherObserved-Valladolid-2018-11-30T07:00:06.00Z",
"type": "WeatherObserved",
"address": {
"type": "Property",
"value": {
  "addressLocality": "Valladolid",
  "addressCountry": "ES",
  "type": "PostalAddress"
}
},
"atmosphericPressure": {
"type": "Property",
"value": 938.9
},
"dataProvider": {
"type": "Property",
"value": "TEF"
},
"dateObserved": {
"type": "Property",
"value": {
  "@type": "DateTime",
  "@value": "2016-11-30T07:00:00.00Z"
}
},
"illuminance": {
"type": "Property",
"value": 1000
},
"location": {
"type": "GeoProperty",
"value": {
  "type": "Point",
  "coordinates": [-4.754444444, 41.640833333]
}
},
"precipitation": {
"type": "Property",
"value": 0
},
"pressureTendency": {
"type": "Property",
"value": 0.5
},
"refDevice": {
"type": "Relationship",
"object": "urn:ngsi-ld:Device:device-0A3478"
},
"relativeHumidity": {
"type": "Property",
"value": 1
},
"snowHeight": {
"type": "Property",
"value": 20
},
"source": {
"type": "Property",
"value": "http://www.aemet.es"
},
"stationCode": {
"type": "Property",
"value": "2422"
},
"stationName": {
"type": "Property",
"value": "Valladolid"
},
"streamGauge": {
"type": "Property",
"value": 50
},
"temperature": {
"type": "Property",
"value": 3.3
},
"uvIndexMax": {
"type": "Property",
"value": 1.0
},
"windDirection": {
"type": "Property",
"value": 135
},
"windSpeed": {
"type": "Property",
"value": 2
}
}'
  3. Get data from quantum-leap: 
    curl --location --request GET 'http://localhost:8668/v2/entities/urn:ngsi-ld:WeatherObserved:Spain-WeatherObserved-Valladolid-2016-11-30T07:00:06.00Z'

Hope that helps:) Best, Stefan

IngMiad commented 2 years ago

Hello @wistefan, Hello @kzangeli,

thank you for your help and the provided example. I can confirm that the proposed example with Docker image 1.0.1 works at my side. I just had to copy the proposed docker-compose.yaml from step 1, create config.yaml from step 2 and run proposed queries from step 3 and 4. All newly created entities will lead to a new notification at QL in step 5 successfully. I also enhanced the example with grafana and can visualize temperature changes from TimescaleDb without any issue. Thank you very much!

I am not sure what made the difference in the end exactly. I can see that the new docker-compose file:

My lessons learned:

"I changed some parts to get it working. I guess you took the compose from one of the tutorials? Can you please send me a link on which one, so that we can update them properly?"

I started with https://github.com/FIWARE/tutorials.Understanding-At-Context/tree/9b84109b104ab7c83da095f85f27e4142138bee0 and continued with information on time series processing following https://user-images.githubusercontent.com/7457382/178001617-0a06e5ae-f27d-4ce7-9302-455e98d40ef6.png. So I expanded my starting example step by step with new components from other tutorial.

If you like you can close the issue. Thanks again,

Inga

kzangeli commented 2 years ago

ok, thank you Inga for all that valuable information.