Is it time to extend this bridge to Helium Network?

[MedadRufus]

In many ways, The Things Network and the Helium network are very similar; End-node hardware can be used for both networks. I am planning a launch of a LoRaWAN tracker transmitting on The Things Network V3 and the Helium network on alternate transmissions. This should increase the coverage a bit, especially in America.

I will take a stab at it in a fork of this repo.

Here is an example packet from the Helium network:

{
  "id": "3e9d7a33-a306-4e25-9db2-5480e7e29967",
  "router_uuid": "48cb0514-4037-4ea1-b0ee-b1ee2078e20c",
  "category": "uplink",
  "sub_category": "uplink_unconfirmed",
  "description": "Unconfirmed data up received",
  "fcnt_up": 6,
  "payload": "IQAAJa6rHt3/qgEAAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rgAAAAAAAL+uAAAAAAAAv64AAAAAAAC/rg==",
  "payload_size": 115,
  "port": 99,
  "reported_at": "1631642657741"
}

Below is the view from the helium console.

[MedadRufus]

Below is another packet(putting it here to aid in creating unittests)

{
  "id": "433fba76-3084-4fbc-be83-e99293d7b1c4",
  "router_uuid": "e6f3d012-18da-44cc-850d-7baa52005ddb",
  "category": "uplink",
  "sub_category": "uplink_unconfirmed",
  "description": "Unconfirmed data up received",
  "fcnt_up": 8,
  "payload": "IQAALamrHt3/uQEAAAAAAADArgAAAAAAAMCuAAAAAAAAwK4AAAAAAADArgAAAAAAAMCuAAAAAAAAwK4AAAAAAADArgAAAAAAAMCuAAAAAAAAwK4AAAAAAADArgAAAAAAAMCuAAAAAAAAwK4AAAAAAADArg==",
  "payload_size": 115,
  "port": 99,
  "reported_at": "1631642706341"
}

[bertrik]

Agreed. I've been playing with Helium a bit in the past weeks. Concepts are basically identical (being LoRaWAN). One of my other projects, a forwarder from TTN to sensor.community to send particulate matter sensor data is just about to get support added.

One thing I'm not completely sure of, if Helium has a usable MQTT server that we can use or you should just use your own.

[MedadRufus]

I am not exactly sure how the MQTT server works for Helium. Agreed. I need to find out.

[bertrik]

I added some classes and unit tests for a Helium message as received on an MQTT server.

My plan is to refactor the code a bit so you can configure a list of (MQTT) sources, for each:

• MQTT URL for example "tcp://eu1.cloud.thethings.network" for TTNv3
• MQTT user name (app name for TTNv3)
• MQTT password (API key for TTNv3)
• MQTT topic, this is typically "v3/+/devices/+/up" for TTNv3 for example, and " helium/+/rx" for Helium for example

I plan to change the format of the configuration file from a properties file to a YAML file, so it is a bit easier to allow lists of items.

[MedadRufus]

Wow thats really fast

[bertrik]

Support has been added, but not actually tested :) Configuration is now with a YAML file. If it does not exist, a new one with reasonable defaults is generated.

The Helium support assumes an MQTT stream, just like TheThingsNetwork. You'll have to set up the MQTT integration for this in the Helium console (and connect it to your application in the 'Flows' section). You will also have to decide about an MQTT server yourself: host your own one, or figure out if you can use another server , e.g. provided by Helium. I have not researched that last part extensively.

If you leave either the Helium or TheThingsNetwork MQTT URL empty in the configuration, that MQTT stream it is not activated. This way you can choose to use Helium, or TheThingsNetwork, or both at the same time.

Gateways from TheThingsNetwork will show up in habitat as "Radio: TheThingsNetwork" and Helium will show up as "Radio: Helium". Helium does not provide an altitude for its gateway locations/antennas, so the habitat "antenna" field will not show altitude for Helium.

This is all a bit untested (except for unit tests), so please report if it works for you (or does not)! I did a very quick regression test for TheThingsNetwork, could not really test it for Helium.

[MedadRufus]

Thats super fast. I will test it

[MedadRufus]

I have it working perfectly with The Things Network. But I simply cannot find out how to make the Helium MQTT integration work. This issue is unrelated to ttnhabbridge anyway but I am trying to figure things out.

On the Helium integration page, I get the MQTT end point details.

I then put the details in the HiveMQ web based MQTT client. Unfortunately, no data from my end node comes through on the MQTT client, although they appear on the Helium console. But very strangely, every hour or so, I see a few messages pop up on the MQTT client. An example is shown below:

{
  "app_eui": "70B3D52DD3000000",
  "decoded": {
    "payload": {
      "batteryVoltage": 3.4000000000000004,
      "brokenSensor": false,
      "childLock": false,
      "highMotorConsumption": false,
      "lowMotorConsumption": false,
      "motorPosition": 536,
      "motorRange": 536,
      "openWindow": false,
      "reason": 1,
      "relativeHumidity": 60.546875,
      "sensorTemperature": 20.5859375,
      "targetTemperature": 17
    },
    "status": "success"
  },
  "dev_eui": "70B3D52DD300088D",
  "devaddr": "65080048",
  "fcnt": 9,
  "hotspots": [
    {
      "channel": 2,
      "frequency": 868.5,
      "hold_time": 1731,
      "id": "11g2xJCHxuFfo5XRwkWJnKy1PvxRcUQMpRssmM7R8iZe9wr9XzK",
      "lat": 55.943804448572,
      "long": -4.338915356277848,
      "name": "icy-watermelon-hare",
      "reported_at": 1634599628627,
      "rssi": -80,
      "snr": 8.199999809265137,
      "spreading": "SF12BW125",
      "status": "success"
    }
  ],
  "id": "8f5bb615-5da5-4045-837a-65becf15b90b",
  "metadata": {
    "adr_allowed": false,
    "cf_list_enabled": false,
    "labels": [
      {
        "id": "8b035567-58f1-49f3-8e1f-cbbb5d83498a",
        "name": "TRV",
        "organization_id": "e8c4bbd7-f83a-4edf-822c-1dad73df7dc9"
      }
    ],
    "multi_buy": 1,
    "organization_id": "e8c4bbd7-f83a-4edf-822c-1dad73df7dc9"
  },
  "name": "perle-room-1",
  "payload": "ARFemxgYIuAA",
  "payload_size": 9,
  "port": 2,
  "reported_at": 1634599628627,
  "uuid": "3455ce47-7bb4-44c1-bfcd-ff6e9ed2c63e"
}

I traced down the gateway name(icy-watermelon-hare) and its in Scotland, and the device name(perle-room-1) matches the name of hotel(Perle Oban Hotel & Spa) in Scotland! So it looks like these MQTT messages are coming from a legitimate source.

View of the MQTT client

I come to 3 conclusions:

1. It looks like one person in Scotland some how managed to get the MQTT integration working.
2. Nobody else seems to have figured out how to use the MQTT integration
3. data security seems to be an issue, since I can see other end node owner's data.

Most public tutorials on integrations seem to only be HTTP based integrations, and hence most support from the Helium developers. MQTT needs some work. I will try to peer into the Helium server code to find out what's going wrong.

[bertrik]

Not sure what is going on, I am not very familiar with HiveMQ. I did some quick testing with my own MQTT server and that seemed to work.

Possibly you are pushing data to some server at the same topic as some other person is doing.

The thing that took me a while to figure out is that I had to configure a "flow" under flows, e.g. drag a device onto the canvas, drag the MQTT integration onto the canvas, and connect them through a line to actually get the data to flow from the device into the integration.

[MedadRufus]

Thanks for the flows heads up! I will try it!

[MedadRufus]

Looks like the flows setup is really important. But now I have run into a integration error. I am using a public broker. Tried with a few different brokers. Error remains. No debug info about this integration error is available elsewhere in the console. Mystery.

[MedadRufus]

Finally got it working. Setting up the flow was important. But I was confused by HiveMQ webbrowser client, which was not receiving mqtt packets and I never got it to work.

Instead, I spun up my own MQTT broker running on a server, keyed in that address info into Helium console Integrations tab, and used the hive-mq's cli to subscribe to the topic:

docker run hivemq/mqtt-cli sub -V 3 -t helium/+/rx -h beta.medadnewman.co.uk -p 1887 -u "medad" -pw "<password>"

and voila! I received MQTT packets:

{
  "id": "dbf4cbbb-2b09-4fd7-83b5-7274de29928c",
  "name": "virtual-device-us",
  "app_eui": "6081F9C2627E358C",
  "dev_eui": "6081F9C2F457D396",
  "devaddr": "18000048",
  "downlink_url": "",
  "fcnt": "0",
  "port": "243",
  "payload": "RKANDw==",
  "reported_at": "1634770613902"
}

I will make a tutorial on this. Its not very straight forward.

[MedadRufus]

Closing because issue has been solved by commit 65f524b1efb2060987029bedc0c9880f33b7cc40. The bridge now works on both the helium network, and Things network, and can be made to receive from both networks at the same time.

[bertrik]

Cool, thanks for testing and making this work!

[LukePrior]

I will make a tutorial on this. Its not very straight forward.

I was just about to come here and ask about Helium support so I would love to see this tutorial when you write it up!

[MedadRufus]

I will prep it this evening or tomorrow

[MedadRufus]

This is a bit of draft tutorial on how to setup Helium MQTT intergration into TTNhabbridge. The key difference between using The Things Network and the Helium Network MQTT intergration is The Things Network hosts their packet broker, while the helium network does not. Therefore, we have to setup our own packet broker, and get Helium to and TTNhabbridge to connect to it. I have here a few steps on how to do it.

1. Setup a MQTT packet broker on machine with port 1883 open to the internet. I prefer to do it with docker. I run the following command, changing the <username> and <secret_password> password fields to something secret :

   docker run \
   --name mosquitto \
   -d \
   -p 1883:1883 \
   -p 9883:9883 \
   -e MQTT_USERNAME=<username> \
   -e MQTT_PASSWORD=<secret_password> \
   cburki/mosquitto:latest

2. Now make Helium to speak with the MQTT broker in the earlier step. To do that, first add a label to your tracker device registration. In this case, I have labeled it icspace26

3. Next in your intergrations tab, make a new MQTT integration. Your endpoint address should be in the format mqtt://<username>:<secret_password>@your-ip-or-domain.com:1883. Your <username> and <secret_password> fields should be the same as the ones your set in step 1.

4. Next, head over to the flows tab and connect the label box with the mqtt intergration box. You should immediately see some data appearing on your MQTT packet broker log. This will be log data from Helium servers connecting with your packet broker.

5. Finally, update the ttnhabbridge.yaml with your MQTT packet broker's address and credentials. Helium will publish incoming messages to your MQTT broker. TTNhabbridge will subscribe to your broker, repackage messages and post it on Habhub. An example ttnhabridge.yaml file looks like this:

   
   ---
   thethingsnetwork:
   url: "tcp://eu1.cloud.thethings.network"
   user: "icss-lora-tracker@ttn"
   pass: "NNSXS.JDUJCMUYUNDVPIOZ53TN6GBTT2P74NBH5FZXYXQ.OCWSRRQ4E46M5OMNATY35GJ6lkaBH6MOWZ5QFP32DMLP823S6M2A"
   topic: "v3/+/devices/+/up"
   helium:
   url: "tcp://beta.medadnewman.co.uk:1887"
   user: "medad"
   pass: "secret_password"
   topic: "helium/+/rx"
   habitat:
   url: "http://habitat.habhub.org"
   timeout: 60
   gwCacheExpirationTime: 600
   payloadEncoding: "cayenne"

[LukePrior]

Wow that's great thanks @MedadRufus.

Might be a good idea to add it to the main wiki?

[LukePrior]

What device and code are using to transmit to both networks @MedadRufus?

I've got a few Heltec and TTGO boards so I was going to see if I can get one of them working.

[bertrik]

@MedadRufus @LukePrior I can add the tutorial text to the README.md or some other file in the archive.

There's also the wiki at https://revspace.nl/TTNHABBridge but I am the only one who can edit that and it's more about the background/design of the software, not so much about the usage.

[MedadRufus]

Yes I think it will be good to put it into the readme of this repo. I will do it soon with a pull request.

[LukePrior]

There's also the wiki at https://revspace.nl/TTNHABBridge but I am the only one who can edit that and it's more about the background/design of the software, not so much about the usage.

This is where I found out most of the information about this project, I see you have a section for Tracker Configuration: https://revspace.nl/TTNHABBridge#Tracker_configuration_for_the_TTN-HAB_bridge_software so maybe it could also be added there?

[MedadRufus]

What device and code are using to transmit to both networks @MedadRufus?

I've got a few Heltec and TTGO boards so I was going to see if I can get one of them working.

Currently I am preparing a tracker using a Murata ABZ radio+MCU module. I have gotten the code to make it transmit on both TTN and helium network. The code may be found here: https://github.com/ImperialSpaceSociety/LoRaMac-node

In theory, if your end-node works on the Things Network, it ought to work on the Helium Network. While its quite hard to get a Helium network compatible gateway, you can test all your device functionality on a Things network compatible gateway. You will need to register your device on either network and hardcode the keys into your end-node code.

My preferred solution for testing the MQTT integration on Helium is to use a Virtual LoraWAN end-node, running entirely in software on my Laptop. Instructions on simulating an end-node can be found on my blog: https://medadnewman.co.uk/2021/10/12/how-to-run-a-virtual-lorawan-end-nodefor-testing/

[Josh-bolt]

This is a bit of draft tutorial on how to setup Helium MQTT intergration into TTNhabbridge. The key difference between using The Things Network and the Helium Network MQTT intergration is The Things Network hosts their packet broker, while the helium network does not. Therefore, we have to setup our own packet broker, and get Helium to and TTNhabbridge to connect to it. I have here a few steps on how to do it.

1. Setup a MQTT packet broker on machine with port 1883 open to the internet. I prefer to do it with docker. I run the following command, changing the <username> and <secret_password> password fields to something secret :

docker run \
    --name mosquitto \
    -d \
    -p 1883:1883 \
    -p 9883:9883 \
    -e MQTT_USERNAME=<username> \
    -e MQTT_PASSWORD=<secret_password> \
    cburki/mosquitto:latest

2. Now make Helium to speak with the MQTT broker in the earlier step. To do that, first add a label to your tracker device registration. In this case, I have labeled it icspace26
3. Next in your intergrations tab, make a new MQTT integration. Your endpoint address should be in the format mqtt://<username>:<secret_password>@your-ip-or-domain.com:1883. Your <username> and <secret_password> fields should be the same as the ones your set in step 1.
4. Next, head over to the flows tab and connect the label box with the mqtt intergration box. You should immediately see some data appearing on your MQTT packet broker log. This will be log data from Helium servers connecting with your packet broker.
5. Finally, update the ttnhabbridge.yaml with your MQTT packet broker's address and credentials. Helium will publish incoming messages to your MQTT broker. TTNhabbridge will subscribe to your broker, repackage messages and post it on Habhub. An example ttnhabridge.yaml file looks like this:

---
thethingsnetwork:
  url: "tcp://eu1.cloud.thethings.network"
  user: "icss-lora-tracker@ttn"
  pass: "NNSXS.JDUJCMUYUNDVPIOZ53TN6GBTT2P74NBH5FZXYXQ.OCWSRRQ4E46M5OMNATY35GJ6lkaBH6MOWZ5QFP32DMLP823S6M2A"
  topic: "v3/+/devices/+/up"
helium:
  url: "tcp://beta.medadnewman.co.uk:1887"
  user: "medad"
  pass: "secret_password"
  topic: "helium/+/rx"
habitat:
  url: "http://habitat.habhub.org"
  timeout: 60
gwCacheExpirationTime: 600
payloadEncoding: "cayenne"

hey , from where you used the docker to run

[Josh-bolt]

What device and code are using to transmit to both networks @MedadRufus? I've got a few Heltec and TTGO boards so I was going to see if I can get one of them working.

Currently I am preparing a tracker using a Murata ABZ radio+MCU module. I have gotten the code to make it transmit on both TTN and helium network. The code may be found here: https://github.com/ImperialSpaceSociety/LoRaMac-node

In theory, if your end-node works on the Things Network, it ought to work on the Helium Network. While its quite hard to get a Helium network compatible gateway, you can test all your device functionality on a Things network compatible gateway. You will need to register your device on either network and hardcode the keys into your end-node code.

My preferred solution for testing the MQTT integration on Helium is to use a Virtual LoraWAN end-node, running entirely in software on my Laptop. Instructions on simulating an end-node can be found on my blog: https://medadnewman.co.uk/2021/10/12/how-to-run-a-virtual-lorawan-end-nodefor-testing/

I'm also trying to send the data to helium console using the esp32, ttgo module. but unfortunately not able to send it, if you direct me to the code for the reference will be helpful

[bertrik]

Can we consider this issue closed with https://github.com/bertrik/ttnhabbridge/commit/858aa956ab1464d65ed326ff56ba85cb3488092b accepted?