ngsi-ld query language: invalid character

fisuda commented 3 years ago

The invalid character error occurs when specifying multibyte characters to a query condition in GET /ngsi-ld/v1/entities. The following is a script to reproduce this issue. A string that has multibyte characters is １２３４.

#!/bin/sh
curl -s http://orion-ld:1026/version

curl -s -iX POST http://orion-ld:1026/ngsi-ld/v1/entityOperations/upsert -H "Content-Type: application/json" \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '[
  {
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "name": {
      "type": "Property",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Property",
      "value": 25
    }
  },
  {
    "id": "urn:ngsi-ld:sensor002",
    "type": "Sensor",
    "name": {
      "type": "Property",
      "value": "５６７８"
    },
    "temperature": {
      "type": "Property",
      "value": 30
    }
  }
]'

curl -s 'http://orion-ld:1026/ngsi-ld/v1/entities?q=name==１２３４' \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' | jq .

The following is a result run it.

{
  "orionld version": "post-v0.6",
  "orion version":   "1.15.0-next",
  "uptime":          "0 d, 0 h, 0 m, 29 s",
  "git_hash":        "nogitversion",
  "compile_time":    "Sun Feb 28 10:19:18 UTC 2021",
  "compiled_by":     "root",
  "compiled_in":     "buildkitsandbox",
  "release_date":    "Sun Feb 28 10:19:18 UTC 2021",
  "doc":             "https://fiware-orion.readthedocs.org/en/master/"
}

HTTP/1.1 204 No Content
Connection: Keep-Alive
Date: Mon, 01 Mar 2021 07:58:14 GMT

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
  "title": "ngsi-ld query language: invalid character",
  "detail": "�"
}

The following is a script for Orion 2.5.2.

#!/bin/sh
curl http://orion:1026/version

curl -iX POST http://orion:1026/v2/op/update -H "Content-Type: application/json" \
--data '{
  "actionType": "append",
  "entities": [
  {
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "name": {
      "type": "Text",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Number",
      "value": 25
    }
  },
  {
    "id": "urn:ngsi-ld:sensor002",
    "type": "Sensor",
    "name": {
      "type": "Text",
      "value": "５６７８"
    },
    "temperature": {
      "type": "Number",
      "value": 30
    }
  }]
}'

curl -s 'http://orion:1026/v2/entities?q=name==１２３４' | jq .

Orion has found an entity that matched the query condition.

{
"orion" : {
  "version" : "2.5.2",
  "uptime" : "0 d, 0 h, 2 m, 57 s",
  "git_hash" : "11e4cbfef30d28347162e5c4ef4de3a5d2797c69",
  "compile_time" : "Thu Dec 17 08:43:22 UTC 2020",
  "compiled_by" : "root",
  "compiled_in" : "43568db1ff93",
  "release_date" : "Thu Dec 17 08:43:22 UTC 2020",
  "doc" : "https://fiware-orion.rtfd.io/en/2.5.2/",
  "libversions": {
     "boost": "1_53",
     "libcurl": "libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0",
     "libmicrohttpd": "0.9.70",
     "openssl": "1.0.2k",
     "rapidjson": "1.1.0",
     "mongodriver": "legacy-1.1.2"
  }
}
}

HTTP/1.1 204 No Content
Connection: Keep-Alive
Fiware-Correlator: 384668ca-7a64-11eb-abad-0242c0a8100d
Date: Mon, 01 Mar 2021 08:00:42 GMT

[
  {
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "name": {
      "type": "Text",
      "value": "１２３４",
      "metadata": {}
    },
    "temperature": {
      "type": "Number",
      "value": 25,
      "metadata": {}
    }
  }
]

$ docker images | grep orion
fiware/orion-ld                          0.7-PRE-73                     bbb310b77759   22 hours ago     978MB
fiware/orion                             2.5.2                          26d39d225f0c   2 months ago     279MB

kzangeli commented 3 years ago

Try using double-quotes for the string (and URL-encode the quotes):

... ?q=name==%22１２３４%22

NGSIv2 and NGSI-LD differs a little here. See chapter 4.9 in the NGSI-LD API spec:

...
ComparableValue = Number / quotedStr / dateTime / date / time
...
quotedStr = String                                                     ; "*char"
...

String shall be a text string as mandated by the JSON Specification, following the ABNF Grammar,
production rule named String, section 7 of IETF RFC 8259 [6].

Spaces are a bit tricky, Hopefully it works both with and without URL-encoding the spaces ...

Let me know how it goes.

fisuda commented 3 years ago

Thank you for your advice. I tried it. It works.

curl -s 'http://orion-ld:1026/ngsi-ld/v1/entities?q=name==%22１２３４%22' \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' | jq .

[
  {
    "@context": "https://context.lab.letsfiware.jp/test-context.jsonld",
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "name": {
      "type": "Property",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Property",
      "value": 25
    }
  }
]

I have a further question. When an attribute name is multibyte characters, how should I specify a query? The case is as follows.

short name

"ｎａｍｅ": {
  "type": "Property",
  "value": "１２３４"
},

IRI

"https://context.lab.letsfiware.jp/dataset#ｎａｍｅ": {
  "type": "Property",
  "value": "１２３４"
},

In the case of q=ｎａｍｅ==%22１２３４%22

{
"type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
"title": "ngsi-ld query language: invalid character",
"detail": "�"
}

In the case of q=https%3A%2F%2Fcontext.lab.letsfiware.jp%2Fdataset%23%82%8E%82%81%82%8D%82%85==%22１２３４%22

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
  "title": "ngsi-ld query language: invalid character",
  "detail": "/"
}

In the case of q=%22ｎａｍｅ%22==%22１２３４%22

{
"type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
"title": "ngsi-ld query language: invalid start token",
"detail": "StringValue"
}

The following is a test script.

#!/bin/sh
curl -s http://orion-ld:1026/version

curl -s -iX POST http://orion-ld:1026/ngsi-ld/v1/entityOperations/upsert -H "Content-Type: application/json" \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '[
  {
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Property",
      "value": 25
    }
  },
  {
    "id": "urn:ngsi-ld:sensor002",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": "５６７８"
    },
    "temperature": {
      "type": "Property",
      "value": 30
    }
  }
]'

curl -s 'http://orion-ld:1026/ngsi-ld/v1/entities?q=ｎａｍｅ==%22１２３４%22' \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' | jq .

{
  "orionld version": "post-v0.6",
  "orion version":   "1.15.0-next",
  "uptime":          "0 d, 0 h, 15 m, 31 s",
  "git_hash":        "nogitversion",
  "compile_time":    "Sun Feb 28 10:19:18 UTC 2021",
  "compiled_by":     "root",
  "compiled_in":     "buildkitsandbox",
  "release_date":    "Sun Feb 28 10:19:18 UTC 2021",
  "doc":             "https://fiware-orion.readthedocs.org/en/master/"
}

HTTP/1.1 204 No Content
Connection: Keep-Alive
Date: Mon, 01 Mar 2021 11:58:14 GMT

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
  "title": "ngsi-ld query language: invalid character",
  "detail": "�"
}

kzangeli commented 3 years ago

When an attribute name is multibyte characters ...

Frankly, I have never tried that. The first thing I would do would be to URL-encode the name (that goes as a URI parameter). And I mean, just try to url-encode ｎａｍｅ- don't add any quotes - that's only for string values (on Right-Hand Side).

kzangeli commented 3 years ago

So, I tried - took me quite some time to figure out how to make curl accept the URL parameter with "funny characters" in it ...

It doesn't work - I will need to "broaden" the set of characters I accept for variable names in 'q'.

This is what I send:

curl localhost:9999/ngsi-ld/v1/entities?prettyPrint=yes -X GET --data-urlencode "q=ｎａｍｅ==1" -G -H "Accept: application/json"

And this is what I get:

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/BadRequestData",
  "title": "ngsi-ld query language: invalid character",
  "detail": "ï"
}

At least, I'm able to reproduce the problem and I can start to try and find a fix! :)

kzangeli commented 3 years ago

I removed the check for forbidden characters in variable names of the 'q' string - seems to be working now. I'll have to consult with my colleagues and boss before I merge this.

kzangeli commented 3 years ago

I decided to merge the PR. Please try it and let me know

fisuda commented 3 years ago

I tried it. The results were as expected. Thank you for your help!

#!/bin/sh
curl -s http://orion-ld:1026/version

echo

curl -s -iX POST http://orion-ld:1026/ngsi-ld/v1/entityOperations/upsert -H "Content-Type: application/json" \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"' \
--data '[
  {
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Property",
      "value": 25
    }
  },
  {
    "id": "urn:ngsi-ld:sensor002",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": 1
    },
    "temperature": {
      "type": "Property",
      "value": 30
    }
  }
]'

curl -s 'http://orion-ld:1026/ngsi-ld/v1/entities?prettyPrint=yes' -X GET --data-urlencode "q=ｎａｍｅ==\"１２３４\"" -G \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"'

echo

curl -s 'http://orion-ld:1026/ngsi-ld/v1/entities?prettyPrint=yes' -X GET --data-urlencode "q=ｎａｍｅ==1" -G \
-H 'Link: <https://context.lab.letsfiware.jp/test-context.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"'

{
  "orionld version": "post-v0.6",
  "orion version":   "1.15.0-next",
  "uptime":          "0 d, 0 h, 43 m, 18 s",
  "git_hash":        "nogitversion",
  "compile_time":    "Mon Mar 1 17:36:28 UTC 2021",
  "compiled_by":     "root",
  "compiled_in":     "buildkitsandbox",
  "release_date":    "Mon Mar 1 17:36:28 UTC 2021",
  "doc":             "https://fiware-orion.readthedocs.org/en/master/"
}

HTTP/1.1 204 No Content
Connection: Keep-Alive
Date: Mon, 01 Mar 2021 21:18:37 GMT

[
  {
    "@context": "https://context.lab.letsfiware.jp/test-context.jsonld",
    "id": "urn:ngsi-ld:sensor001",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": "１２３４"
    },
    "temperature": {
      "type": "Property",
      "value": 25
    }
  }
]

[
  {
    "@context": "https://context.lab.letsfiware.jp/test-context.jsonld",
    "id": "urn:ngsi-ld:sensor002",
    "type": "Sensor",
    "ｎａｍｅ": {
      "type": "Property",
      "value": 1
    },
    "temperature": {
      "type": "Property",
      "value": 30
    }
  }
]

$ docker images | grep orion-ld
fiware/orion-ld                               0.7-PRE-74                     49fb0b1b6858   4 hours ago     978MB

FIWARE / context.Orion-LD

ngsi-ld query language: invalid character #737