Arcgis sink. Avoid transforming `null` into `"null"`

danielvillalbamota commented 1 month ago

Problem

Arcgis sink is wrongly mapping null to string "null" which is a problem when feature layer field is type "number" where arcgis API response with 400 code. It is also a problem with the rest of types if a service consume data where a value has "null" as string.

I have configured a custom notif with ngsi patching and when one of the attribute does not exist in the entity the macro substitution set the value to null.

{
  "url": "http://host-cygnus/notify",
  "ngsi": {
    "type": "fiwareid",
    "humedad_relativa": {
      "type": "Number",
      "value": "${relativeHumidityValue}"
    },
    "nombre": {
      "type": "Text",
      "value": "${gis_id}"
    }
  }
}

Cygnus receive the null value properly:

time=2024-06-04T10:21:19.825Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=aggregate | msg=com.telefonica.iot.cygnus.sinks.NGSIArcgisFeatureTableSink$NGSIArcgisAggregator[483] : [NGSIArcgisAggregator] aggregate - Selected context ->{"id":"W01","type":"fiwareid","isPattern":"null","attributes":[{"name":"humedad_relativa","type":"Number","value":null,"metadatas":[]},{"name":"nombre","type":"Text","value":null,"metadatas":[]}]}

But when cygnus sink makes de query the replace the null by "null":

time=2024-06-04T10:21:19.912Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e-abbb74ef6e59 | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=requestHTTP | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.RestApi[179] : requesting HTTP: https://arcgis_server/server/rest/services/OP/OP_Plataform/FeatureServer/157/updateFeatures?outSR=%7B%22wkid%22%3A4326%7D&token=oPgtq...PGprrvqYBscFXWW6fNq&rollbackOnFailure=true---METHOD: POST---BODY: features=%5B%7B%22geometry%22%3A%7B%22x%22%3A0.0%2C%22y%22%3A0.0%2C%22spatialReference%22%3A%7B%22wkid%22%3A4326%7D%7D%2C%22attributes%22%3A%7B%22OBJECTID%22%3A5%2C%22fiwareid%22%3A%22W01%22%2C%22humedad_relativa%22%3A%22null%22%2C%22nombre%22%3A%22null%22%7D%7D%5D&f=json

Followed by these traces:

time=2024-06-04T10:21:20.014Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=requestHTTP | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.RestApi[263] : Disposing connection objects
time=2024-06-04T10:21:20.014Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=sendFeatureList | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.RestFeatureTable[269] : Response code: 200
time=2024-06-04T10:21:20.015Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=checkHttpResponse | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.RestApi[374] : Response has erros, 400 Response Error 400  Unable to complete operation.
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=checkResponse | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.RestFeatureTable[235] : Error: 400
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=commitFeatures | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.ArcgisFeatureTable[549] : Error: 400
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=setError | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.ArcgisFeatureTable[596] : Error Message: Error: 400
time=2024-06-04T10:21:20.015Z | lvl=DEBUG | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=commitFeatures | msg=com.telefonica.iot.cygnus.backends.arcgis.restutils.ArcgisFeatureTable[553] : pendingFeatures.commitFeatures adding listener to commitFeatureFuture: 1
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=persistAggregation | msg=com.telefonica.iot.cygnus.sinks.NGSIArcgisFeatureTableSink[403] : [arcgis-sink-no-nm] Error persisting batch, ArcgisException - Error: 400
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=persistBatch | msg=com.telefonica.iot.cygnus.sinks.NGSIArcgisFeatureTableSink[310] : [arcgis-sink-no-nm] Error persisting batch, CygnusRuntimeError.CygnusRuntimeError. Error: 400
time=2024-06-04T10:21:20.015Z | lvl=ERROR | corr=afa03cc3-c9cf-4573-923e; cbnotif=1 | trans=802847b3-6961-48be-ad4e | srv=N/A | subsrv=N/A | comp=cygnus-ngsi | op=processNewBatches | msg=com.telefonica.iot.cygnus.sinks.NGSISink[610] : CygnusRuntimeError. CygnusRuntimeError. Error: 400

Expected behavior

The POST request to arcgis should map the null value without quotes.

fgalan commented 1 month ago

PR https://github.com/telefonicaid/fiware-cygnus/pull/2377

Waiting for @danielvillalbamota validation before closing this issue.

fgalan commented 1 month ago

Additional fix in PR https://github.com/telefonicaid/fiware-cygnus/pull/2378

danielvillalbamota commented 3 weeks ago

It works. I close the issue

telefonicaid / fiware-cygnus

Arcgis sink. Avoid transforming `null` into `"null"` #2376

Problem

Expected behavior