Telemetry timeseries needs to contain all the key-value pairs

[Sylphe88]

Using the MQTT extension, I'm unable to read telemetry data if the JSON string does not contain all the entries from the mqtt-config.json file. Conf file extract example:

"timeseries": [
              {
                "type": "double",
                "key": "Temperature",
                "value": "${$.temp}"
              },
          {
                "type": "double",
                "key": "Voltage",
                "value": "${$.volt}"
              }
            ]

If I send {"volt":8.14, "temp":23.9} (I can shuffle KV pairs), it works, the gateway and the TB dashboard both update. However, if I want to send individual KVs, like {"volt":2.27}, the gateway will reject the message and won't route it to TB.

Is there anything I'm doing wrong? It seems like the gateway mapper has to find all the registered KVs before it aknowledges a MQTT message.

The gateway reports the following error when it fails to decode the previous JSON:

2018-02-12 06:55:40,673 [MQTT Call: cbd029dd-0b71-4e65-88aa-78e535fc2a0d] INFO  o.t.g.e.m.c.l.MqttTelemetryMessageListener - [cRIO/abcd/Telemetry] Failed to decode message: [123, 34, 118, 111, 108, 116, 34, 58, 48, 46, 50, 52, 125]
java.lang.RuntimeException: Failed to apply expression $.temp
    at org.thingsboard.gateway.util.converter.AbstractJsonConverter.apply(AbstractJsonConverter.java:77)
    at org.thingsboard.gateway.util.converter.AbstractJsonConverter.eval(AbstractJsonConverter.java:48)
    at org.thingsboard.gateway.extensions.mqtt.client.conf.mapping.MqttJsonConverter.getKvEntries(MqttJsonConverter.java:114)
    at org.thingsboard.gateway.extensions.mqtt.client.conf.mapping.MqttJsonConverter.parse(MqttJsonConverter.java:100)
    at org.thingsboard.gateway.extensions.mqtt.client.conf.mapping.MqttJsonConverter.convert(MqttJsonConverter.java:84)
    at org.thingsboard.gateway.extensions.mqtt.client.listener.MqttTelemetryMessageListener.messageArrived(MqttTelemetryMessageListener.java:42)
    at org.eclipse.paho.client.mqttv3.internal.CommsCallback.deliverMessage(CommsCallback.java:469)
    at org.eclipse.paho.client.mqttv3.internal.CommsCallback.handleMessage(CommsCallback.java:380)
    at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:184)
    at java.lang.Thread.run(Thread.java:748)

[ashvayka]

Hi @Sylphe88

We will make this more user-friendly in the new release. For now, you can use the following workaround: Use filter expression in your mapping and map items individually: "$[?(@.volt)]" should filter items that have volt. Similar for temp.

You can create two elements in "mapping" array each one will use different filters (one for volt and one for temp) and push data items individually.

@mp-loki Please provide sample of mqtt configuration and update this issue with PR that contains fix for this case.

[aderusha]

Have others been able to make use of this workaround? If I put filters into place I get rid of the errors but I also get rid of any telemetry being picked up in the filtered mappings, even when the filter matches.

edit: I take that back - it does pull values, but only the last item in the list of elements, and only if that last item exists. For example, using the configuration below, TB IoT Gateway will only report temperature and will only do so if temperature exists.

{
  "brokers": [
    {
      "host": "us-west.thethings.network",
      "port": 1883,
      "ssl": false,
      "retryInterval": 3000,
      "credentials": {
        "type": "basic",
        "username": "ttn-example-app",
        "password": "ttn-account-v2._xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"        
      },
      "mapping": [
        {
          "topicFilter": "+/devices/+/up",
          "converter": {
            "type": "json",
            "filterExpression": "$[?(@.payload_fields.humidity)]",
            "deviceNameJsonExpression": "${$.dev_id}",
            "timeseries": [
              {
                "type": "long",
                "key": "humidity",
                "value": "${$.payload_fields.humidity}"
              }
            ]
          }
        },
        {
          "topicFilter": "+/devices/+/up",
          "converter": {
            "type": "json",
            "filterExpression": "$[?(@.payload_fields.temperature)]",
            "deviceNameJsonExpression": "${$.dev_id}",
            "timeseries": [
              {
                "type": "double",
                "key": "temperature",
                "value": "${$.payload_fields.temperature}"
              }
            ]
          }
        }
      ]
    }
  ]
}

[arskanov]

Has this progressed anywhere? I'm attempting to send timeseries values one-by-one (http), but I tb-gateway expects all of the "timeseries" array's key/values to be present in the json.

        "timeseries": [
          {
            "type": "double",
            "key": "temperature",
            "value": "${$.data.temperature}",
            "transformer": {
              "type": "intToDouble"
            }
          },
          {
            "type": "double",
            "key": "humidity",
            "value": "${$.data.humidity}",
            "transformer": {
              "type": "intToDouble"
            }
          }
        ]

with

{
    "time": "1523597296",
    "device": "216660",
    "lat": "60.0", 
    "lng": "22.0",
    "data": {
        "temperature": "4865.0"
    }
}

results in

java.lang.RuntimeException: Failed to apply expression $.data.humidity
    at org.thingsboard.gateway.util.converter.AbstractJsonConverter.apply(AbstractJsonConverter.java:77)
    at org.thingsboard.gateway.util.converter.AbstractJsonConverter.eval(AbstractJsonConverter.java:48)
    at org.thingsboard.gateway.extensions.sigfox.conf.mapping.SigfoxDeviceDataConverter.getKvEntries(SigfoxDeviceDataConverter.java:68)
    at org.thingsboard.gateway.extensions.sigfox.conf.mapping.SigfoxDeviceDataConverter.parseBody(SigfoxDeviceDataConverter.java:51)

The filters mentioned above seem to be meant for MQTT only, if I understand. Are there any workarounds for optional k/v pairs in the JSON, or would sending a "magic" value and using a custom isApplicable method be the best solution for now?

[arskanov]

This issue for MQTT seems to be solved at least in this commit.

I'm running 1.2.1 due to getting timeout's on connect to Thingsboard on higher versions. For the Sigfox HTTP api, I essentially copied the same changes from that commit, but the file I modified was SigfoxDeviceDataConverter.java instead of MQTTJsonConverter. The changes to AbstractJsonConverter were identical.

Now I can receive key/value pairs optionally!

[mp-loki]

@arskanov Right, there was a software fix for this in 1.2.2 & 1.4.0. Can you please share the details about the timeout connect problem? Please create another issue for that and provide your gateway configuration files. Meanwhile, I'll close this issue

[kaizen8501]

Hi~ @Sylphe88 Did you solve problem?

I also operate to send individual KVs like as below.

• Topic : v1/usr/openhouse/down

• Payload : {"cmdId":11809,"humidity":["2018-05-14 13:05:53",35.2]}

• Topic : v1/usr/openhouse/down

• Payload : {"cmdId":11809,"temperature":["2018-05-14 13:05:44",33.2]

So I writed mqtt-config.json as below.

  "brokers": [
    {   
      "host": "test.sktiot.com",
      "port": 1883,
      "ssl": false,
      "retryInterval": 3000,
      "credentials": {
        "type": "basic",
        "username": "xxxx",
        "password": "xxxx"
      },  
      "mapping": [
        {
          "topicFilter": "v1/usr/openhouse/down",
          "converter": {
            "type": "json",
            "filterExpression": "$[?(@.humidity)]",
            "deviceNameJsonExpression": "${$.cmdId}",
            "deviceTypeJsonExpression": "SF_TempHumi",
            "attributes": [
            ],
            "timeseries": [
              {
                "type": "double",
                "key": "humidity",
                "value": "${$.humidity[1]}"
              }
            ]
          }
        },
        {
          "topicFilter": "v1/usr/openhouse/down",
          "converter": {
            "type": "json",
            "filterExpression": "$[?(@.temperature)]",
            "deviceNameJsonExpression": "${$.cmdId}",
            "deviceTypeJsonExpression": "SF_TempHumi",
            "timeseries": [
              {
                "type": "double",
                "key": "temperature",
                "value": "${$.temperature[1]}"
              }
            ]
          }
        }
      ],
      ...
      ...

[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - kaizen json message: {"cmdId":11809,"temperature":["2018-05-14 13:25:27",33.2]}
[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - kaizen Data before filtering {"cmdId":11809,"temperature":["2018-05-14 13:25:27",33.2]}
[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - kaizen Data after filtering [{"cmdId":11809,"temperature":["2018-05-14 13:25:27",33.2]}], $[?(@.temperature)

[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.l.MqttTelemetryMessageListener - v1/usr/openhouse/down,{"cmdId":11809,"humidity":["2018-05-14 13:05:53",35.2]}
[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - json message: {"cmdId":11809,"humidity":["2018-05-14 13:05:53",35.2]}
[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - Data before filtering {"cmdId":11809,"humidity":["2018-05-14 13:05:53",35.2]}
[MQTT Call: d5681eba-9f33-4148-8939-354ece59e6e2] INFO  o.t.g.e.m.c.c.m.MqttJsonConverter - Data after filtering [], **$[?(@.temperature)]**

I used TB-Gateway 1.4.1

But I think It has some problem because TB-Gateway only use last "filterExpression". Is there anything I'm doing wrong? Anyone please help me and give a guide.

Thank you.

[Sylphe88]

@kaizen8501 In my case the tb-gateway 1.2.1 (and onward) worked. Maybe we should wait for @ashvayka 's answer in case there's something different going on with 1.4.x

[rof20004]

I am with same problem.

[mp-loki]

@rof20004 please provide your configuration and explain what is the problem (better in new ticket). The problem with the above configuration is that there are two entries with the same topic filter, thus the first one gets overridden with the second.

[arthurborgesdev]

And what if the message doesn't have any json payload? (But only raw values?) How do I parse it using mappings?

So far my message is like this:

Topic: AWPLC Gateway B8:27:EB:93:40:40/MEDIDORES_211/Kwh[1]/READ Payload: 2.72727