TTN with custom format

[filgiuff]

I've registered a device on The things network , it sends measures and I see them on the app server TTN:

{
  "distance": 5377,
  "humidity": 53.97,
  "pressure": 5397,
  "temperature": 3.969999999999999
}

I've tried to do the provisioning of this device on the IoTagent-LoRaWAN:

{

    "devices": [
        {
            "device_id": "00aa10a1088354aa",
            "service": "test",
            "service_path": "/test_issues",
            "entity_name": "00aa10a1088354aa",
            "entity_type": "Device",
            "transport": "LORA",
            "attributes": [
                {
                    "object_id": "distance",
                    "name": "distance",
                    "type": "number"
                },
                {
                    "object_id": "humidity",
                    "name": "humidity",
                    "type": "number"
                },
                {
                    "object_id": "pressure",
                    "name": "pressure",
                    "type": "number"
                },
                {
                    "object_id": "temperature",
                    "name": "temperature",
                    "type": "number"
                }
            ],
            "lazy": [],
            "commands": [],

            "internal_attributes": {
                "lorawan": {
                    "application_server": {
                        "host": "eu.thethings.network",
                        "provider": "TTN",
                        "username": "fakeusername",
                        "password": "fakettn-account-aaaaaaaaaaaaaaaaaaaaaaa"
                    },
                    "app_eui": "fakeappeui",
                    "application_id": "fakeappid",
                    "application_key": "fakeappkey",
                    "data_model": "application_server",
                    "dev_eui": "00aa10a1088354aa"
                }
            },
            "protocol": "LORAJSON"
        }
    ]
}

it creates the entity on Orion and the Mqtt topic is subscribed correctly:

 iotagent-lora_1    | {"timestamp":"2019-10-21T16:44:12.740Z","level":"info","message":"Subscribing to MQTT topic:fakeappid/devices/00aa10a1088354aa/up"}
 iotagent-lora_1   | {"timestamp":"2019-10-21T16:44:12.741Z","level":"info","message":"Mqtt topic subscribed:fakeappid/devices/00aa10a1088354aa/up"}

But the values of the measures are empty

{
    "id": "00aa10a1088354aa",
    "type": "Device",
    "TimeInstant": {
        "type": "ISO8601",
        "value": " ",
        "metadata": {}
    },
    "dataformat_protocol": {
        "type": "Text",
        "value": "LORAJSON",
        "metadata": {}
    },
    "distance": {
        "type": "number",
        "value": " ",
        "metadata": {}
    },
    "humidity": {
        "type": "number",
        "value": " ",
        "metadata": {}
    },
    "pressure": {
        "type": "number",
        "value": " ",
        "metadata": {}
    },
    "temperature": {
        "type": "number",
        "value": " ",
        "metadata": {}
    },
    "transport_protocol": {
        "type": "Text",
        "value": "LORA",
        "metadata": {}
    }
}

When a new measure arrives on TTN I haven't any line on the IoTAgent log.

My question:

1) Is the "application_server" data model currently supported ? The official documentation just links CayenneLpp and CBOR, instead github says also "The payload is directly decoded by the LoRaWAN application server."

2) Am I making any mistakes in provisioning?

[dcalvoalonso]

Hi @filgiuff,

I have done the following tests and it has worked properly:

1. Provision a new device in the IoT Agent using application_server in data_model property :

curl -X POST \
  http://localhost:4061/iot/devices \
  -H 'Content-Type: application/json' \
  -H 'fiware-service: atosioe' \
  -H 'fiware-servicepath: /lorattn' \
  -d '{
  "devices": [
    {
      "device_id": "lora_n_003",
      "entity_name": "LORA-N-003",
      "entity_type": "LoraDevice",
      "timezone": "America/Santiago",
      "attributes": [
        {
          "name": "barometric_pressure_0",
          "type": "Number"
        },
        {
          "name": "digital_in_3",
          "type": "Number"
        },
        {
            "name": "temperature_1",
            "type": "Number" 
        }
      ],
      "internal_attributes": {
        "lorawan": {
          "application_server": {
            "host": "eu.thethings.network",
            "username": "",
            "password": "",
            "provider": "TTN"
          },
          "dev_eui": "",
          "app_eui": "",
          "application_id": "",
          "application_key": "",
          "data_model": "application_server"
        }
      },
      "protocol": "LORAJSON"
    }
  ]
}'

2. Configure TTN console to use a Custom Payload Format. I have already added a dummy Decoder function that will return always the same payload just for testing:

   function Decoder(bytes) {
   
   return {
   barometric_pressure_0: 99,
   temperature_1: 43,
   digital_in_3: 1
   }
   }

3. Simulate uplink data using TTN. It does not matter which data is sent, the dummy decoder function returns always the same payload. This is correctly decoded by TTN.

4. The IoT Agent receives the update from TTN and sends the request to the Orion CB as you can see:

{"timestamp":"2019-10-23T06:03:56.706Z","level":"info","message":"New message in topic"}
time=2019-10-23T06:03:56.706Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.MongoDBDeviceRegister | srv=n/a | subsrv=n/a | msg=Looking for device with id [lora_n_003]. | comp=IoTAgent
time=2019-10-23T06:03:56.709Z | lvl=ERROR | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Releasing [MONGO-ALARM] | comp=IoTAgent
{"timestamp":"2019-10-23T06:03:56.710Z","level":"info","message":"IOTA provisioned devices:"}
time=2019-10-23T06:03:56.710Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=executeWithDeviceInfo entityName LORA-N-003 type LoraDevice apikey  attributes [{"name":"barometric_pressure_0","type":"Number","value":99},{"name":"digital_in_3","type":"Number","value":1},{"name":"temperature_1","type":"Number","value":43}] deviceInformation {"lazy":[],"active":[{"name":"barometric_pressure_0","type":"Number","object_id":"barometric_pressure_0"},{"name":"digital_in_3","type":"Number","object_id":"digital_in_3"},{"name":"temperature_1","type":"Number","object_id":"temperature_1"}],"commands":[],"staticAttributes":[],"subscriptions":[],"_id":"5dafec43b58ab34b61c67f13","creationDate":"2019-10-23T05:59:31.310Z","id":"lora_n_003","type":"LoraDevice","name":"LORA-N-003","service":"atosioe","subservice":"/lorattn","internalId":null,"internalAttributes":{"lorawan":{"application_server":{"host":"eu.thethings.network","username":"","password":"","provider":"TTN"},"dev_eui":"","app_eui":"","application_id":"","application_key":"","data_model":"application_server"}},"protocol":"LORAJSON"} | comp=IoTAgent
time=2019-10-23T06:03:56.710Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.MongoDBGroupRegister | srv=n/a | subsrv=n/a | msg=Looking for group params ["type"] with queryObj {"type":"LoraDevice"} | comp=IoTAgent
time=2019-10-23T06:03:56.712Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.MongoDBGroupRegister | srv=n/a | subsrv=n/a | msg=Device group for fields [["type"]] not found: [{"type":"LoraDevice"}] | comp=IoTAgent
time=2019-10-23T06:03:56.712Z | lvl=ERROR | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.Alarms | srv=n/a | subsrv=n/a | msg=Raising [MONGO-ALARM]: {"name":"DEVICE_GROUP_NOT_FOUND","message":"Couldn\t find device group","code":404} | comp=IoTAgent
time=2019-10-23T06:03:56.713Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=error {"name":"DEVICE_GROUP_NOT_FOUND","message":"Couldn\t find device group","code":404} in get group device | comp=IoTAgent
time=2019-10-23T06:03:56.713Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=typeInformation {"lazy":[],"active":[{"name":"barometric_pressure_0","type":"Number","object_id":"barometric_pressure_0"},{"name":"digital_in_3","type":"Number","object_id":"digital_in_3"},{"name":"temperature_1","type":"Number","object_id":"temperature_1"}],"commands":[],"staticAttributes":[],"subscriptions":[],"_id":"5dafec43b58ab34b61c67f13","creationDate":"2019-10-23T05:59:31.310Z","id":"lora_n_003","type":"LoraDevice","name":"LORA-N-003","service":"atosioe","subservice":"/lorattn","internalId":null,"internalAttributes":{"lorawan":{"application_server":{"host":"eu.thethings.network","username":"","password":"","provider":"TTN"},"dev_eui":"","app_eui":"","application_id":"ari_ioe_app_demo1","application_key":"","data_model":"application_server"}},"protocol":"LORAJSON"} | comp=IoTAgent
time=2019-10-23T06:03:56.713Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=typeInformation: {"lazy":[],"active":[{"name":"barometric_pressure_0","type":"Number","object_id":"barometric_pressure_0"},{"name":"digital_in_3","type":"Number","object_id":"digital_in_3"},{"name":"temperature_1","type":"Number","object_id":"temperature_1"}],"commands":[],"staticAttributes":[],"subscriptions":[],"_id":"5dafec43b58ab34b61c67f13","creationDate":"2019-10-23T05:59:31.310Z","id":"lora_n_003","type":"LoraDevice","name":"LORA-N-003","service":"atosioe","subservice":"/lorattn","internalId":null,"internalAttributes":{"lorawan":{"application_server":{"host":"eu.thethings.network","username":"","password":"","provider":"TTN"},"dev_eui":"","app_eui":"","application_id":"ari_ioe_app_demo1","application_key":"","data_model":"application_server"}},"protocol":"LORAJSON"} | comp=IoTAgent
time=2019-10-23T06:03:56.714Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=Updating device value in the Context Broker at [http://localhost:1026/v2/entities/LORA-N-003/attrs?type=LoraDevice] | comp=IoTAgent
time=2019-10-23T06:03:56.714Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=Using the following request:

{
    "url": "http://localhost:1026/v2/entities/LORA-N-003/attrs?type=LoraDevice",
    "method": "POST",
    "headers": {
        "fiware-service": "atosioe",
        "fiware-servicepath": "/lorattn"
    },
    "json": {
        "barometric_pressure_0": {
            "type": "Number",
            "value": 99,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T06:03:56.714Z"
                }
            }
        },
        "digital_in_3": {
            "type": "Number",
            "value": 1,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T06:03:56.714Z"
                }
            }
        },
        "temperature_1": {
            "type": "Number",
            "value": 43,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T06:03:56.714Z"
                }
            }
        },
        "TimeInstant": {
            "type": "DateTime",
            "value": "2019-10-23T06:03:56.714Z"
        }
    }
}

 | comp=IoTAgent
time=2019-10-23T06:03:56.719Z | lvl=DEBUG | corr=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | trans=e319ec01-a072-4e88-b3c1-eb5fc52ff049 | op=IoTAgentNGSI.NGSIService | srv=n/a | subsrv=n/a | msg=Received the following response from the CB: Value updated successfully
 | comp=IoTAgent
{"timestamp":"2019-10-23T06:03:56.719Z","level":"info","message":"Observations sent to CB successfully for device "}

5. Send a request to the Orion CB to retrieve the entity:

curl -X GET \
  http://localhost:1026/v2/entities \
  -H 'fiware-service: atosioe' \
  -H 'fiware-servicepath: /lorattn'

The answer contains the values:

[
    {
        "id": "LORA-N-003",
        "type": "LoraDevice",
        "TimeInstant": {
            "type": "DateTime",
            "value": "2019-10-23T05:59:42.00Z",
            "metadata": {}
        },
        "barometric_pressure_0": {
            "type": "Number",
            "value": 99,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T05:59:42.00Z"
                }
            }
        },
        "digital_in_3": {
            "type": "Number",
            "value": 1,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T05:59:42.00Z"
                }
            }
        },
        "temperature_1": {
            "type": "Number",
            "value": 43,
            "metadata": {
                "TimeInstant": {
                    "type": "DateTime",
                    "value": "2019-10-23T05:59:42.00Z"
                }
            }
        }
    }
]

[dcalvoalonso]

In order to help you:

1. Which version of the IoT Agent are you using? Is it a Docker deployment or from sources?
2. Which is the content of your configuration file?
3. Which version of the Context Broker are you using?
4. Could you please share some information about the decoder function and a payload example in hex format?

[filgiuff]

1. I'm using v1.2.3, with docker, my image has been created 4 month ago, immediately after the release 1.2.3
2. my config-docker.js file

   
   var config = {};

config.iota = { /**

• Configures the log level. Appropriate values are: FATAL, ERROR, INFO, WARN and DEBUG. */ logLevel: 'DEBUG',

  /**

• When this flag is active, the IoTAgent will add the TimeInstant attribute to every entity created, as well

• as a TimeInstant metadata to each attribute, with the current timestamp. */ timestamp: true,

  /**

• Context Broker configuration. Defines the connection information to the instance of the Context Broker where

• the IoT Agent will send the device data. */ contextBroker: { /**

  • Host where the Context Broker is located. */ host: 'iotstack_orion',

    /**

  • Port where the Context Broker is listening. */ port: '1026',

    /**

  • Version of NGSI

  • ngsiVersion: 'v2' */

  },

  /**

• Configuration of the Northbound server of the IoT Agent. */ server: { /**

  • Port where the IoT Agent will be listening for requests. */ port: 8095 },

  /**

• Default resource of the IoT Agent. This value must be different for every IoT Agent connecting to the IoT

• Manager. */ defaultResource: '/iot/d',

  /**

• Defines the configuration for the Device Registry, where all the information about devices and configuration
• groups will be stored. There are currently just two types of registries allowed:
• • 'memory': transient memory-based repository for testing purposes. All the information in the repository is
• wiped out when the process is restarted.
• • 'mongodb': persistent MongoDB storage repository. All the details for the MongoDB configuration will be read

• from the 'mongoDb' configuration property. */ deviceRegistry: { type: 'mongodb' },

  /**

• Mongo DB configuration section. This section will only be used if the deviceRegistry property has the type

• 'mongodb'. */ mongodb: { /**

  • Host where MongoDB is located. If the MongoDB used is a replicaSet, this property will contain a

  • comma-separated list of the instance names or IPs. */ host: 'iotstack_mongodb',

    /**

  • Port where MongoDB is listening. In the case of a replicaSet, all the instances are supposed to be listening

  • in the same port. */ port: '27017',

    /**

  • Name of the Mongo database that will be created to store IOTAgent data. */ db: 'iotagentlora'

    /**

  • Name of the set in case the Mongo database is configured as a Replica Set. Optional otherwise. */ // replicaSet: '' },

  /**

• Types array for static configuration of services. Check documentation in the IoTAgent Library for Node.js for
• further details:

• https://github.com/telefonicaid/iotagent-node-lib#type-configuration */ types: {},

  /**

• Default service, for IOTA installations that won't require preregistration. */ service: 'howtoService',

  /**

• Default subservice, for IOTA installations that won't require preregistration. */ subservice: '/howto',

  /**

• URL Where the IOTA Will listen for incoming updateContext and queryContext requests (for commands and passive

• attributes). This URL will be sent in the Context Registration requests. */ providerUrl: 'http://iotstack_iotagent-lora:8097',

  /**

• Default maximum expire date for device registrations. */ deviceRegistrationDuration: 'P1Y',

  /**

• Default type, for IOTA installations that won't require preregistration. */ defaultType: 'Thing' };

module.exports = config;

3. My Orion version is 1.13.0 (see [this related issue](https://github.com/Atos-Research-and-Innovation/IoTagent-LoRaWAN/issues/64) )

4. I've just used your example above and also this one doesn't work :(

[filgiuff]

I've found the error: in all my previous cases the Device ID on TTN were equals to the DeviceEUI and I used the DeviceEUI on the device_id field of the agent. Changing to correct Device ID it works fine. This should be better documented. Thank you for the support.