search

telefonicaid / fiware-cygnus

A connector in charge of persisting context data sources into other third-party databases and storage systems, creating a historical view of the context
https://fiware-cygnus.rtfd.io/
GNU Affero General Public License v3.0
64 stars 104 forks source link

Unknown value: x-ngsiv2-normalized-compacted for NGSI format #2102

Open jason-fox opened 2 years ago

jason-fox commented 2 years ago

Problem reported in the last TSC where status of Cygnus was followed up. Apparently, Cygnus doesn’t know how to process part of the header sent with the notification. Not sure if this a Cygnus fix or an Orion-LD fix.

NGSI-v2 with Orion - works as expected

Using fiware/cygnus-ngsi

curl 'http://localhost:5080/v1/version'
{"success":"true","version":"2.10.0.5bb41dfcca1e25db664850e6b7806e3cf6a2aa7b"}

Using fiware/orion

curl  'http://localhost:1026/version/'
{
"orion" : {
  "version" : "3.2.0",
  "uptime" : "0 d, 0 h, 2 m, 7 s",
  "git_hash" : "2a51a86b4a672f1eb59a69539d763275b8581e73",
  "compile_time" : "Tue Sep 7 18:06:44 UTC 2021",
  "compiled_by" : "root",
  "compiled_in" : "90cc3acfe3e2",
  "release_date" : "Tue Sep 7 18:06:44 UTC 2021",
  "machine" : "x86_64",
  "doc" : "https://fiware-orion.rtfd.io/en/3.2.0/",
  "libversions": {
     "boost": "1_66",
     "libcurl": "libcurl/7.61.1 OpenSSL/1.1.1g zlib/1.2.11 nghttp2/1.33.0",
     "libmosquitto": "2.0.11",
     "libmicrohttpd": "0.9.70",
     "openssl": "1.1",
     "rapidjson": "1.1.0",
     "mongoc": "1.17.4",
     "bson": "1.17.4"
  }
}
}

Open up the supermarket, switch on the lamp & etc. Create a v2 subscription on Cygnus MySQL port using attrsFormat: normalized

curl -iX POST \
  'http://localhost:1026/v2/subscriptions' \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: openiot' \
  -H 'fiware-servicepath: /' \
  -d '{
  "description": "Notify Cygnus MySQL of all context changes",
  "subject": {
    "entities": [
      {
        "idPattern": ".*"
      }
    ]
  },
  "notification": {
    "http": {
      "url": "http://cygnus:5050/notify"
    }
  },
  "throttling": 5
}'

Verify subscription

 curl -iX GET \ 
  'http://localhost:1026/v2/subscriptions' \
  -H 'fiware-service: openiot'  
[
  {
    "id": "617689a22f8cc268aa225429",
    "description": "Notify Cygnus MySQL of all context changes",
    "status": "active",
    "subject": {
      "entities": [
        {
          "idPattern": ".*"
        }
      ],
      "condition": {
        "attrs": []
      }
    },
    "notification": {
      "timesSent": 4,
      "lastNotification": "2021-10-25T10:41:10.000Z",
      "attrs": [],
      "onlyChangedAttrs": false,
      "attrsFormat": "normalized",
      "http": {
        "url": "http://cygnus:5050/notify"
      },
      "lastSuccess": "2021-10-25T10:41:10.000Z",
      "lastSuccessCode": 200
    },
    "throttling": 5
  }
]

NGSI-v2 with Orion-LD in backwards compatibility mode - not working

Using fiware/cygnus-ngsi

curl 'http://localhost:5080/v1/version'
{"success":"true","version":"2.10.0.5bb41dfcca1e25db664850e6b7806e3cf6a2aa7b"}

Using fiware/orion-ld

curl  'http://localhost:1026/version/'
{
  "orionld version": "0.8.1",
  "orion version":   "1.15.0-next",
  "uptime":          "0 d, 0 h, 6 m, 1 s",
  "git_hash":        "nogitversion",
  "compile_time":    "Sun Jun 6 08:13:51 UTC 2021",
  "compiled_by":     "root",
  "compiled_in":     "buildkitsandbox",
  "release_date":    "Sun Jun 6 08:13:51 UTC 2021",
  "doc":             "https://fiware-orion.readthedocs.org/en/master/"
}

Open up the farm, start the tractor & etc. Create a v2 subscription on Cygnus MySQL port using attrsFormat: x-ngsiv2-normalized-compacted

curl -L -X POST 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
-H 'Content-Type: application/ld+json' \
-H 'NGSILD-Tenant: openiot' \
-H 'Cookie: _csrf=MAPTGFPcoPnewsGCWklHi4Mq' \
--data-raw '{
  "description": "Notify Cygnus MySQL of all context changes",
  "type": "Subscription",
  "entities": [{"type": "Device"}],
  "watchedAttributes": ["heartRate"],
  "notification": {
    "attributes": ["heartRate"],
    "format": "x-ngsiv2-normalized-compacted",
    "endpoint": {
      "uri": "http://cygnus:5050/notify",
      "accept": "application/json"
    }
  },
   "@context": "http://context/ngsi-context.jsonld"
}'
curl -L -X POST 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
-H 'Content-Type: application/ld+json' \
-H 'NGSILD-Tenant: openiot' \
--data-raw '{
  "description": "Notify  Screen  of all context changes",
  "type": "Subscription",
  "entities": [{"type": "Device"}],
  "watchedAttributes": ["heartRate"],
  "notification": {
    "attributes": ["heartRate"],
    "format": "x-ngsiv2-normalized-compacted",
    "endpoint": {
      "uri": "http://iot-sensors:3000/subscription/low-stock-farm001-ngsiv2",
      "accept": "application/json"
    }
  },
   "@context": "http://context/ngsi-context.jsonld"
}'

Verify subscriptions

curl -L -X GET 'http://localhost:1026/ngsi-ld/v1/subscriptions/' \
-H 'NGSILD-Tenant: openiot'

NGSI-LD Payload is x-ngsiv2-normalized-compacted which is valid NGSI-v2

{
 "subscriptionId": "urn:ngsi-ld:Subscription:61769080785d37c448cf420b",
 "data": [
  {
   "id": "urn:ngsi-ld:Device:cow001",
   "type": "Device",
   "heartRate": {
    "type": "Property",
    "value": 52,
    "metadata": {
     "unitCode": "5K",
     "observedAt": "2021-10-25T11:14:33.000Z"
    }
   }
  }
 ]
}

Cygnus Log

ime=2021-10-25T11:14:54.114Z | lvl=WARN | corr=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | trans=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | srv=openiot | subsrv=/ | comp=cygnus-ngsi | op=getEvents | msg=com.telefonica.iot.cygnus.handlers.NGSIRestHandler[345] : Unknown value: x-ngsiv2-normalized-compacted for NGSI format
time=2021-10-25T11:14:54.114Z | lvl=WARN | corr=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | trans=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | srv=openiot | subsrv=/ | comp=cygnus-ngsi | op=doPost | msg=org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet[237] : Received bad request from client. 
org.apache.flume.source.http.HTTPBadRequestException: x-ngsiv2-normalized-compacted format not supported
    at com.telefonica.iot.cygnus.handlers.NGSIRestHandler.getEvents(NGSIRestHandler.java:346)
    at org.apache.flume.source.http.HTTPSource$FlumeHTTPServlet.doPost(HTTPSource.java:235)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:535)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:15
95)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:564)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:673)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:591)
    at java.lang.Thread.run(Thread.java:748)
time=2021-10-25T11:14:54.114Z | lvl=DEBUG | corr=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | trans=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | srv=openiot | subsrv=/ | comp=cygnus-ngsi | op=sendResponse | msg=org.eclipse.jetty.server.HttpChannel[693] : sendResponse info=null content=HeapByteBuffer@31330f83[p=0,l=0,c=0,r=0]={<<<>>>} complete=true committing=true callback=Blocker@4ce3bbc1{null}
time=2021-10-25T11:14:54.114Z | lvl=DEBUG | corr=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | trans=04a86b3c-2ca8-44cd-84a4-71c0a4356929 | srv=openiot | subsrv=/ | comp=cygnus-ngsi | op=commit | msg=org.eclipse.jetty.server.HttpChannel[747] : COMMIT for /notify on HttpChannelOverHttp@80d07a3{r=1,c=true,a=DISPATCHED,uri=//cygnus:5050/notify}
400 Bad request from client. x-ngsiv2-normalized-compacted format not supported HTTP/1.1
Date: Mon, 25 Oct 2021 11:14:54 GMT
Cache-Control: must-revalidate,no-cache,no-store
jason-fox commented 2 years ago

The crux of the issue is should Orion-LD spoof the attrsFormat back to normalized or should Cygnus accept a wider range of acceptable formats (i.e. allowing x-ngsiv2-normalized-compacted = normalized) or both?

@mapedraza @kzangeli

mapedraza commented 2 years ago

Since the data is processed by NGSIv2 endpoint, it makes sense that the values allowed for attrsFormat are aligned with the NGSIv2 specification ( attrsFormat= normalized)

In addition, according to the NGSIv2 specification (section Notification Messages), we can see only specified values are allowed.

Notifications must include the Ngsiv2-AttrsFormat HTTP header with the value of the format of the associated subscription, so that notification receivers are aware of the format without needing to process the notification payload."

Note that Cygnus only process notifications in normalized mode, keyValues or custom does not have sense

kzangeli commented 2 years ago

So, sure, Orion-LD could use the older names of "normalized" and "keyValues". These would imply expanded values for attribute names and entity types.

For compacted values, we add "-compact" ?

jason-fox commented 2 years ago

For compacted values, we add "-compact" ?

  • "normalized-compact"
  • "keyValues-compact" ?

I would say NO - normalized is just normalized regardless of whether it is expanded or compacted. An NGSI-v2 notification has no concept of "compact" since it doesn't deal with linked data JSON-LD compaction. so just call of all of the available x-ngsiv2- compatible formats with the headers Ngsiv2-AttrsFormat=normalized or Ngsiv2-AttrsFormat=keyValues only.

With an NGSI-v2 notification you are not dealing with @context any more.

kzangeli commented 2 years ago

ok, we kind of lose valuable info there. However, that info is part of the subscription, so, I guess it's not quite lost.

If all are OK with this, I'll modify Orion-LD accordingly-

fgalan commented 2 years ago

If all are OK with this, I'll modify Orion-LD accordingly-

So, no action in Cygnus? Can this issue be closed here?

jason-fox commented 2 years ago

Looks like an Orion-LD change, but leave it open until we can confirm that it is fixed.