New Controller: LORA Direct

[enesbcs]

I am currently testing a new node-to-node communication form, LORA direct. (no WAN)

@TD-er I am suggesting the following data structures:

struct Lora_SysInfoStruct
{
  byte header = 255;
  byte ID = 1;
  byte sourcelUnit;
  byte mac[6];
  word build;
  char Name[25];
  byte nodeType;
};

struct Lora_SensorInfoStruct
{
  byte header = 255;
  byte ID = 3;
  byte sourcelUnit;
  byte destUnit;
  byte destTaskIndex;
  byte sensorType;
  char taskName[26];
  char ValueNames[VARS_PER_TASK][26];
};

struct Lora_SensorDataStruct
{
  byte header = 255;
  byte ID = 5;
  byte sourcelUnit;
  byte destUnit;
  byte destTaskIndex;
  float Values[VARS_PER_TASK];
};

struct Lora_CommandDataStruct
{
  byte header = 255;
  byte ID = 7;
  byte sourcelUnit;
  byte destUnit;
  char commandline[64];
};

I've written a test-sketch for my BsFrance LoRa32u4 module based on the old ArduinoEasy code which i will publish soon. (this device has 2k ram + 32k flash)

The RPI is able to receive packages without problem, Duty cycle is handled at the sender side, in this case by the LoRa32u4, the RPI is a bridge-like device in the setup.

Referenced to: https://github.com/enesbcs/rpieasy/issues/75

[TD-er]

Have you seen my recent PR regarding TTN? https://github.com/letscontrolit/ESPEasy/pull/2546

Of this PR, especially have a look at the JavaScript decoder functions which will be executed on the TTN server. In short, i have added very basic support for all plugins to send out the standard 1 .. 4 float values, but will add more support for plugins to output more values in a compact format specific for that plugin. See the raw encoder functions for Sysinfo and GPS:

https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/src/_P026_Sysinfo.ino#L147-L169

https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/src/_P082_GPS.ino#L523-L544

I am not sure if you can run JS code on the Pi, but if so we can maintain the same decoder files for both platforms. If for some reason a plugin will later support more variables, it can be filtered on packet length or we may add a version byte to its stream. Currently the raw stream does start with the plugin_id, which is a very easy filter for decoding it.

For LoRa, as well as LoRaWAN, there are limits on how long your air time can be. So you want to make sure the data stream is as short as possible and also unambiguous when decoding it. That last part may be a bit difficult for some plugins where the user may select what value to output. So that's why I came up with this RAW packed format.

I was planning to also add support for these kind of raw packed streams into the ESPeasy p2p network, so we can use it also among nodes, regardless of their network or platform.

See also the (incomplete) documentation for this.

[enesbcs]

Sure Javascript can be executed from inside Python with the Js2Py package. I see this TTN thing, but i prefer (local) direct communication forms instead third party server (internet) integrations. They are technically interesting but not so practical solutions. The raw streams can be useful for LoRa, ESPNOW and either Bluetooth P2P communcation in the future.

[TD-er]

I see this TTN thing, but i prefer (local) direct communication forms instead third party server (internet) integrations. They are technically interesting but not so practical solutions.

I never said it should be done solely via TTN (by the way you can also connect your own server to process the data yourself. It is then still encrypted over the internet) It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

The raw streams can be useful for LoRa, ESPNOW and either Bluetooth P2P communcation in the future.

That I totally agree upon and that's also why I put so much effort to add support for these RAW packed encoder thingies. They can be re-used for a lot of things. (pun intended :) ) Also for storage on SPIFFS in large binary dumps. (See the cache controller I'm also working on)

[enesbcs]

It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

I see. But how can we use javascript encoders/decoders in Arduino on device that has 2KiB RAM and 32KiB flash for example?

[TD-er]

It is just that TTN requires the decoders to be handling data in a certain way and if we want to make our own, why not borrow their ideas to make our work simpler so we can use the same decoders for different platforms?

I see. But how can we use javascript encoders/decoders in Arduino on device that has 2KiB RAM and 32KiB flash for example?

We don't. But for those we could try to add a decoder to the same plugin. It is just that having a generic all-in-one decoder to process all plugins is very practical to have on some more powerful nodes like R'pi or TTN servers etc. And just for those purposes it is potentially saving a lot of work if those decoders could be shared among platforms. For the small nodes like the LoRa32u4, you probably are not interested in decoding all possible plugins, but only a handful. So for those it will need some C++ decoding. But still it would be nice if we could also share those encoder/decoder codes in some way. (not sure how)

[TD-er]

In general the small nodes will only supply data from a sensor, so they just encode the data. Only if a node does need to receive sensor data and also needs to decode it (instead of just forwarding it via another medium) then we need to have a decoder written for that platform.

[TD-er]

If you want to do direct LoRa communication from node to node, you still need to add some headers indicating who's sending, who has to receive it and what the payload is. So the raw packed stream can be a good part of the payload, which format can be shared among platforms. But you still need to do some coding on the node itself to process it.

It is just my hope that we can come up with some kind of generic way of generating and interpreting the payload per plugin, so we can add new ones on various platforms without a lot of work. Each platform still needs to have its own wrappers to process the payload, but that only has to be done once per platform/medium. The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

That's my Utopian idea, but I am not sure it is achievable like this.

[enesbcs]

I am using currently the following (working) sketch with the LoRa32u4 for simple sending. https://github.com/enesbcs/EasyLoraSensor Currently the whole thing uses 1KiB RAM and 20KiB flash space.

Sending performed by the LoraDirect.ino in the same style as the ESPEasy P2P UDP networking works. Names only sent once at startup for minimizing traffic later, then the Task values sent at the requested intervals, if the Duty Cycle lets it.. Currently all four task value is sent, but can be modified to respect "ValueCount". On the receiver side (RPI) the received data stored in Dummy tasks without much thinking, as there are no real device beneath these tasks, they simply receive and forward values to other Controllers.

I am not sure what is the benefit of this decoders? https://github.com/letscontrolit/ESPEasy/blob/mega/misc/TTN/packed_converter.js

[enesbcs]

The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

What is the concrete payload structure that you recommend?

[TD-er]

I don't think starting with the existing p2p implementation is a bad idea (in fact it is a good idea to get started) But the p2p layer is rather inefficient and limits the number of values to 4. So that's why I later will add a new packet definition for the p2p protocol which carries the raw packed payload.. That's just a new ID in the p2p header so it will remain compatible with the old versions, just adding a new option that will only be able to be handled by newer builds. So this introduction of my LoRa/TTN efforts is just to make sure that you will not spend a lot of time thinking about the same stuff and come up with almost the same solution but incompatible :)

About the the "converters" part. I used it in my development to "clean up" parts that were not yet decoded. So initially it was just 80+ times the same code cut-and-pasted and then stripped out values that were not available on some plugins. Just see it as a left-over to-do list of mine. In the end we probably do not need the "converter" part anymore, or at least not with entries for all plugins, since the bulk of the decoding part is done in the decoders.js file.

[TD-er]

The Payload structure should be done only once for all platforms and the payload encoding/content should be done only once per plugin.

What is the concrete payload structure that you recommend?

This is the description of the header: https://github.com/letscontrolit/ESPEasy/blob/1a1b7ce219e793f4087748037209531b4df3de87/misc/TTN/packed_decoder.js#L272-L288

• Plugin_id : 1 byte
• IDX: 2 bytes (same IDX entry as mainly used for Domoticz, only now 2 bytes)
• samplesetcount : 1 byte (see documentation on "sample set initiator"
• valuecount : 1 byte

And then the binary packed data stream of the plugin. This header should be enough to help decode that.

[TD-er]

If for some reason the single byte of plugin ID is no longer enough, we can start with a special byte to signal the 2-byte plugin ID. (e.g. set the first bit high)

But please feel free to suggest changes. Right now this code is probably running on a handful of nodes and even less TTN accounts :)

[enesbcs]

* Plugin_id : 1 byte
* IDX: 2 bytes (same IDX entry as mainly used for Domoticz, only now 2 bytes)
* samplesetcount : 1 byte (see [documentation on "sample set initiator"](https://espeasy.readthedocs.io/en/latest/Controller/_Controller.html?highlight=sample%20set#controller-parameters)
* valuecount : 1 byte

And then the binary packed data stream of the plugin.

All right then you are suggesting something similar:

struct Lora_SensorDataStruct
{
 byte plugin_id;
 word idx;
 byte samplesetcount;
 byte valuecount;
 float Values[VARS_PER_TASK]; // valuecount * float
};

But in Raw LoRa communication there are no addressing (it has to be included in the payload), this is why i am using the old ESPEasy P2P packets, as they almost perfect... This is why i think that we can not treat a high level TTN/LoraWAN protocol payload as a simple RAW BLE/LoRa/ESPNOW payload. In LoRa we need at least: 1 byte for sender address, 1 byte for destination address (with some reserved addresses for example 0 is broadcast and 1 for default gateway)

The idx approach is an interesting one, so at the receiver side the user has to create manually the device with this IDX to be able to receive? The old approach sending the DestinationTaskIndex is a little simpler i think. (creating automatically the task at the specific taskindex if the slot is free)

Sending valuecount is a nice idea, using this field i can use one generic struct without decoding stuff, only changing the buffer length to include the necessary ones at the end. :)

And what about the 1st introduction message which may include Nodename, Node MAC for generating a list of nodes? It will be a nice feature to know what nodes was active and when...

[TD-er]

I think we're using not the same definition for "payload" here :)

I consider the "Raw packed stream" as payload. Everything else what's needed is just a wrapper (header) to get the data sent via the medium. So if you need something to address another node, that's just transport layer/medium dependent and thus part of the header or wrapper or whatever term is needed. It does not contain the actual payload data, but is just needed to get the data somewhere.

LoRaWAN is also sending quite a bit more data, but that's not of interest to use the data.

The IDX is already an existing flag which can be sent along with the data.

Like I said it is used mainly in Domoticz to distinguish what sensor is used. (not the sensor type, but the unique entry in the Domoticz sensor list) We can also use it to interpret or identify the data on the receiving end, but for now it is optional. For LoRaWAN it is known what the sender is, but for your application the sender should be included in the wrapper/header. (it is not part of the payload) For example when multiple sensors of the same type are being used on a single node, it can be useful to identify sensor data. For example "temp cooling" and "temp oil".

About the value count, if you look at the decoder I made, you will see almost all plugins except Sysinfo and GPS now have a very basic decoder. It is a not efficient encoder sending integer values which are the result of a float multiplied by 10'000. So on the decoding end you need to divide the value by 10'000 and thus get a float with 4 decimals.

But since the plugin knows best how to send information tightly packed, I added the option to let a plugin generate a raw packed structure. So the Sysinfo plugin manages to send 11 values in just 23 bytes.

The first introduction message is not yet included in the LoRaWAN/TTN plugin. The used must do some stuff and I don't know (yet) what LoRaWAN/TTN does support regarding auto discovery of nodes. As soon as that's known, then it may be (very) useful to add some auto discovery to the controller.

The Homie controller does have some auto discovery features and I know some home automation systems like Home Assistant are also working hard on making the discovery as simple as possible.

[enesbcs]

word plugin_id might needed... Controller is almost done, still testing between my Lora32u4 and Raspberry.

[TD-er]

16 bit you mean?

[enesbcs]

yes, i mean a 16bit word variable.

[TD-er]

Well, I think we can use 1 byte for all controllers up-to 7 bits (127) and use the MSB to identify the 2-byte plugins. We can correct the bit later in the decoder. So in pseudo code:

if (plugin_id <= 127) {
  // 1 byte ID

} else {
  return (32768 + id) & 0xFFFF;
}

[enesbcs]

Lora32u4 sketches can be found here: https://github.com/enesbcs/EasyLora Lora part and structures are here: https://github.com/enesbcs/EasyLora/blob/master/EasyLoraSensor/LoraDirect.ino

[enesbcs]

Experimental support added in commit https://github.com/enesbcs/rpieasy/commit/40f6867cad8d04404bbdf1dc5ea2f0128c3b22b5

[enesbcs]

Closed as no feedback provided. Feel free to reopen in case of related questions.

[TD-er]

I am now looking into the TTN part of sending data. (LoRaWAN/Things network) And I was wondering what should we do with plugin IDs? Right now it is limited to 8 bits and I'm not even sure all places where it is being used is having uint8_t as type, so we may also run into issues when passing 127.

Should we go for 16 bit? And are we going to run into other limits like these? (e.g. unit numbers) Should we discuss this in another issue (and on what repo :) )?

[enesbcs]

https://github.com/letscontrolit/ESPEasy/issues/2661