Tight Integration with The Things Network

The "The Things Network" import panel should ask for; Zone, Application, API Key. Poll Interval to be selected. Upon submitting that, The Things Network should be queried for all the devices in that zone/application, and listed with checkbox selector so one can choose which devices to be imported. Poll Interval can be overridden from the above.

For each selected device, another request to TTN to retrieve the latest packets, and get hold of the "$.upload_message.decoded_payload" object, which contains the data points that we might want. Again with selection of those, and each selected one can have an overridden Poll Interval, then plus Scaling and Unit.

Quite complex, but can be very useful to capture customers who use Things Network.

To access The Things Network, the following two URLs are most interesting;

Devices Api; https://{{zone}}.cloud.thethings.network/api/v3/applications/{{application_id}}/devices

Storage Api; https://{{zone}}.cloud.thethings.network/api/v3/as/applications/{{application_id}}/devices/{{device_id}}/packages/storage/uplink_message

The Authorization Key is via the http header Authorization: Bearer {{authkey}}

Tests can be made with; Zone; eu1 Application; l2w-test-app Auth Key; NNSXS.6SND4GB4FKKJMWSMAR3PPX7TUPTWBLV7UOLCQMY.QAJ6D57MSZH4KXXOOV2NJYT3CWPEDG2UG7ORWDV2PXFMC63H2JDA

One important aspect is that there are more than one so called f_port and that needs to be configured to Sensetif. So when reading packets, it is important to both show thef_port and let the user select packets for a given f_port number. (On the devices that are on the l2w-test-app then f_port=2)

With the following request;

curl -H "Authorization: Bearer NNSXS.6SND4GB4FKKJMWSMAR3PPX7TUPTWBLV7UOLCQMY.QAJ6D57MSZH4KXXOOV2NJYT3CWPEDG2UG7ORWDV2PXFMC63H2JDA" https://eu1.cloud.thethings.network/api/v3/applications/l2w-test-app/devices

one should get something like this;

{
  "end_devices": [
    {
      "ids": {
        "device_id": "eui-a84041bdf1844fb8",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "A84041BDF1844FB8",
        "join_eui": "A840410000000101"
      },
      "created_at": "2022-02-05T07:21:12.890Z",
      "updated_at": "2022-02-05T07:22:53.177Z"
    },
    {
      "ids": {
        "device_id": "lt-22222-1",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "A840414C21828CE0",
        "join_eui": "A000000000000101"
      },
      "created_at": "2021-03-25T14:47:46.454Z",
      "updated_at": "2021-04-05T09:13:40.150Z"
    },
    {
      "ids": {
        "device_id": "mcf88-1",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "70B3D58FF1013F32",
        "join_eui": "A000000000000101"
      },
      "created_at": "2021-04-05T11:35:14.532Z",
      "updated_at": "2021-07-26T09:48:35.942Z"
    },
    {
      "ids": {
        "device_id": "room1",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "A84041D46182A02C",
        "join_eui": "A000000000000100"
      },
      "created_at": "2021-03-31T08:01:28.267Z",
      "updated_at": "2021-07-26T10:04:28.248Z"
    },
    {
      "ids": {
        "device_id": "room2",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "A84041B4B182A030",
        "join_eui": "A000000000000100"
      },
      "created_at": "2021-03-31T08:34:02.476Z",
      "updated_at": "2021-07-26T10:01:51.157Z"
    },
    {
      "ids": {
        "device_id": "room3",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "A840415781822FB2",
        "join_eui": "A84041DB11822FB2"
      },
      "created_at": "2021-03-31T08:37:13.144Z",
      "updated_at": "2021-07-26T09:48:23.478Z"
    },
    {
      "ids": {
        "device_id": "vicki1",
        "application_ids": {
          "application_id": "l2w-test-app"
        },
        "dev_eui": "70B3D52DD3003D7A",
        "join_eui": "70B3D52DD3000000"
      },
      "created_at": "2022-04-09T08:22:36.218Z",
      "updated_at": "2022-04-09T08:53:52.728Z"
    }
  ]
}

And requesting

curl -H "Authorization: Bearer NNSXS.6SND4GB4FKKJMWSMAR3PPX7TUPTWBLV7UOLCQMY.QAJ6D57MSZH4KXXOOV2NJYT3CWPEDG2UG7ORWDV2PXFMC63H2JDA" https://eu1.cloud.thethings.network/api/v3/as/applications/l2w-test-app/devices/room1/packages/storage/uplink_message

will bring back a stream of json packets, separated by LineFeed. NOTE: Not a valid single json document.

Showing the last 3 results;

{"result":{"end_device_ids":{"device_id":"room1","application_ids":{"application_id":"l2w-test-app"},"dev_eui":"A84041D46182A02C","dev_addr":"260B9298"},"received_at":"2022-09-14T08:21:47.507618867Z","uplink_message":{"f_port":2,"f_cnt":13743,"frm_payload":"y/sIDAJEAX//f/8=","decoded_payload":{"battery":3.067,"bytes":[203,251,8,12,2,68,1,127,255,127,255],"humidity":"58.0","sensor":"LHT65","temperature_DS":"327.67","temperature_SHT":"20.60"},"decoded_payload_warnings":["Niclas:{\"bytes\":[203,251,8,12,2,68,1,127,255,127,255],\"fPort\":2}"],"rx_metadata":[{"gateway_ids":{"gateway_id":"bali-automation-lab","eui":"A840411F6B104150"},"time":"2022-09-14T08:21:47.275877Z","timestamp":1515548987,"rssi":-63,"channel_rssi":-63,"snr":11.2,"location":{"latitude":55.8973393382053,"longitude":13.140238523483278,"altitude":55,"source":"SOURCE_REGISTRY"},"channel_index":4,"received_at":"2022-09-14T08:21:47.276607490Z"}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867300000","timestamp":1515548987,"time":"2022-09-14T08:21:47.275877Z"},"received_at":"2022-09-14T08:21:47.303986633Z","consumed_airtime":"0.061696s","version_ids":{"brand_id":"dragino","model_id":"lht65","hardware_version":"_unknown_hw_version_","firmware_version":"1.8","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"eu1","cluster_address":"eu1.cloud.thethings.network"}}}}
{"result":{"end_device_ids":{"device_id":"room1","application_ids":{"application_id":"l2w-test-app"},"dev_eui":"A84041D46182A02C","dev_addr":"260B9298"},"received_at":"2022-09-14T08:41:47.550192442Z","uplink_message":{"f_port":2,"f_cnt":13744,"frm_payload":"y/sIDAJEAX//f/8=","decoded_payload":{"battery":3.067,"bytes":[203,251,8,12,2,68,1,127,255,127,255],"humidity":"58.0","sensor":"LHT65","temperature_DS":"327.67","temperature_SHT":"20.60"},"decoded_payload_warnings":["Niclas:{\"bytes\":[203,251,8,12,2,68,1,127,255,127,255],\"fPort\":2}"],"rx_metadata":[{"gateway_ids":{"gateway_id":"bali-automation-lab","eui":"A840411F6B104150"},"time":"2022-09-14T08:41:47.309316Z","timestamp":2715586660,"rssi":-64,"channel_rssi":-64,"snr":9.2,"location":{"latitude":55.8973393382053,"longitude":13.140238523483278,"altitude":55,"source":"SOURCE_REGISTRY"},"channel_index":6,"received_at":"2022-09-14T08:41:47.311118597Z"}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867700000","timestamp":2715586660,"time":"2022-09-14T08:41:47.309316Z"},"received_at":"2022-09-14T08:41:47.336676999Z","consumed_airtime":"0.061696s","version_ids":{"brand_id":"dragino","model_id":"lht65","hardware_version":"_unknown_hw_version_","firmware_version":"1.8","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"eu1","cluster_address":"eu1.cloud.thethings.network"}}}}
{"result":{"end_device_ids":{"device_id":"room1","application_ids":{"application_id":"l2w-test-app"},"dev_eui":"A84041D46182A02C","dev_addr":"260B9298"},"received_at":"2022-09-14T09:01:47.592950915Z","uplink_message":{"f_port":2,"f_cnt":13745,"frm_payload":"y/sICgJHAX//f/8=","decoded_payload":{"battery":3.067,"bytes":[203,251,8,10,2,71,1,127,255,127,255],"humidity":"58.3","sensor":"LHT65","temperature_DS":"327.67","temperature_SHT":"20.58"},"decoded_payload_warnings":["Niclas:{\"bytes\":[203,251,8,10,2,71,1,127,255,127,255],\"fPort\":2}"],"rx_metadata":[{"gateway_ids":{"gateway_id":"bali-automation-lab","eui":"A840411F6B104150"},"time":"2022-09-14T09:01:47.356430Z","timestamp":3915624419,"rssi":-66,"channel_rssi":-66,"snr":9.8,"location":{"latitude":55.8973393382053,"longitude":13.140238523483278,"altitude":55,"source":"SOURCE_REGISTRY"},"channel_index":3,"received_at":"2022-09-14T09:01:47.358103237Z"}],"settings":{"data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},"coding_rate":"4/5","frequency":"867100000","timestamp":3915624419,"time":"2022-09-14T09:01:47.356430Z"},"received_at":"2022-09-14T09:01:47.383907407Z","consumed_airtime":"0.061696s","version_ids":{"brand_id":"dragino","model_id":"lht65","hardware_version":"_unknown_hw_version_","firmware_version":"1.8","band_id":"EU_863_870"},"network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"eu1","cluster_address":"eu1.cloud.thethings.network"}}}}

And inside there are the f_port and decoded_payload. decoded_payload contains the names of interest. This is totally user defined, and potentially could be full JSON objects, rather than primitive values, but let's require only primitive values initially, i.e. IF there is an Object, show the name, but disallow selection on it, and the REASON in hover-over/tooltip.

I imagine this;

Select Zone (one of eu1 (Europe), nam1 (North America) or au1 (Australia)), application, poll interval, and if each device should become its own subsystem, or be added to an existing subsystem and device+variable merged in the datapoint name with an underscore ("_")
Fetch Devices with a Button press, and fill a list to the left with device names.
Select which devices should be part of the Import. For each device, be able to select "Device Template" or create a new one (see below). Maybe default is "No Template", in which case the device will be left out of the import.
Validate; Fetch stored data and make sure that the Template matches the devices selected.
For each Device, for each variable in template, create a data point definition, either by creating a subsystem per device and adding datapoints in that subsystem with the same variable names, or creating datapoints with a concatenated name (device + "_" + varName) in the selected subsystem. If subsystems are to be created, create subsystems first, then wait 2 seconds and continue with the data points.

Create Device Template popup(?); Fetch the stored data for the device, present the variable names in a list to the left, and the min, max, unit and scaling. (I have not decided on storage period yet. Probably going to remove it from datapoints and always use "account wide" storage periods). Template also needs a name.

I think it is important to minimize "waiting" in UI thread.

Sensetif / sensetif-app-plugin

Tight Integration with The Things Network #29