Received payload is different from "send" payload

[itofficeeu]

Thanks for a awesome library!

During testing sending data to the backend, I always send the same small set of data.

`
buffer[0] = (char) 9; buffer[1] = (char) 29;

Serial.print("celciusInt"); Serial.println(celciusInt);
Serial.print("buffer[0]="); Serial.println(buffer[0]);
Serial.print("buffer[1]="); Serial.println(buffer[1]);
Serial.print("sizeof(buffer)="); Serial.println(sizeof(buffer));

// Prepare upstream data transmission at the next possible time.
//LMIC_setTxData2(1, lpp.getBuffer(), lpp.getSize(), 0);

// Send on port 1, without asking for confirmation:
LMIC_setTxData2(1, buffer, sizeof(buffer), 0); // 0x0929 for 23.45

`

At TTN I get totally different data in the payload. The data aren't even the same each time I send data. They change each time they are send. The 'funny' thing is, they make a series of data for each restart of the microcontroller.

Take a look at this:

Restart Number 1 2 3 ...

Payload in TTN

       count
           0           BF90    BF90    BF90    ...
           1           3581    3581    3581    ...
           2           4DC7    4DC7    4DC7    ...
           3           7CC9    7CC9    7CC9    ...
          ...          ...

Are the data mixed with the count?...

Am I doing something wrong?

Thanks :)

[matthijskooijman]

Now that's weird... Are you using OTAA or ABP? And you're just using the regular TTN backend?

My first guess would be invalid encryption keys, which could lead to incorrect decryption. However, if you would configure the wrong keys, then the checksum and other header data would also be messed up and data would never reach your application in TTN.

Can you run with debug level 2 (see config.h in the library) and paste the results here? Perhaps there's a hint in there...

[itofficeeu]

HW: Heltect WiFi LoRa 32

Using ABP.

Backend is TTN.

Debug/Monitor - now reduced to fit code below

Starting...
RXMODE_RSSI
LMIC setup done!
celciusInt2345
buffer[0]=9
buffer[1]=29
sizeof(buffer)=2
6600: engineUpdate, opmode=0x808
6617: Uplink data pending
6654: Considering band 0, which is available at 6589
6947: Considering band 3, which is available at 0
7224: No channel found in band 3
7408: Considering band 0, which is available at 6589
7701: No channel found in band 0
7886: Considering band 1, which is available at 6589
8179: Airtime available at 6589 (channel duty limit)
8472: Ready for uplink
8623: Updating info for TX at 6617, airtime will be 2896. Setting available time for band 1 to 296217
9182: TXMODE, freq=868100000, len=15, SF=7, BW=125, CR=4/5, IH=0
Packet queued
----
----
72149: irq: dio: 0x0 flags: 0x8
72161: Scheduled job 0x3ffc3b88, cb 0x400d1ea8 ASAP
72168: Running job 0x3ffc3b88, cb 0x400d1ea8, deadline 0
72243: Scheduled job 0x3ffc3b88, cb 0x400d1bb0 at 134618
134618: Running job 0x3ffc3b88, cb 0x400d1bb0, deadline 134618
134744: RXMODE_SINGLE, freq=868100000, SF=7, BW=125, CR=4/5, IH=0
135084: irq: dio: 0x1 flags: 0x80
135094: Scheduled job 0x3ffc3b88, cb 0x400d33d0 ASAP
135119: Running job 0x3ffc3b88, cb 0x400d33d0, deadline 0
135439: Scheduled job 0x3ffc3b88, cb 0x400d1c00 at 197406
197406: Running job 0x3ffc3b88, cb 0x400d1c00, deadline 197406
197533: RXMODE_SINGLE, freq=869525000, SF=9, BW=125, CR=4/5, IH=0
198832: irq: dio: 0x1 flags: 0x80
198842: Scheduled job 0x3ffc3b88, cb 0x400d3400 ASAP
198847: Running job 0x3ffc3b88, cb 0x400d3400, deadline 0
198945: EV_TXCOMPLETE (includes waiting for RX windows)
199254: Scheduled job 0x3ffc24bc, cb 0x400d0cfc at 824253
199573: engineUpdate, opmode=0x900
824253: Running job 0x3ffc24bc, cb 0x400d0cfc, deadline 824253
celciusInt2345
buffer[0]=9
buffer[1]=29
sizeof(buffer)=2
824264: engineUpdate, opmode=0x908
824434: Uplink data pending
824591: Considering band 0, which is available at 6589
824895: Considering band 3, which is available at 0
825182: No channel found in band 3
825378: Considering band 0, which is available at 6589
825682: No channel found in band 0
825877: Considering band 1, which is available at 296217
826191: Considering band 2, which is available at 6589
826495: No channel found in band 2
826691: Considering band 1, which is available at 296217
827005: Airtime available at 296217 (channel duty limit)
827320: Airtime available at 399312 (global duty limit)
827629: Ready for uplink
827786: Updating info for TX at 824434, airtime will be 2896. Setting available time for band 1 to 1114034
828376: TXMODE, freq=868300000, len=15, SF=7, BW=125, CR=4/5, IH=0
Packet queued
----
831283: irq: dio: 0x0 flags: 0x8
831294: Scheduled job 0x3ffc3b88, cb 0x400d1ea8 ASAP
831300: Running job 0x3ffc3b88, cb 0x400d1ea8, deadline 0
831391: Scheduled job 0x3ffc3b88, cb 0x400d1bb0 at 893752
893752: Running job 0x3ffc3b88, cb 0x400d1bb0, deadline 893752
893879: RXMODE_SINGLE, freq=868300000, SF=7, BW=125, CR=4/5, IH=0
894218: irq: dio: 0x1 flags: 0x80
894228: Scheduled job 0x3ffc3b88, cb 0x400d33d0 ASAP
894253: Running job 0x3ffc3b88, cb 0x400d33d0, deadline 0
894573: Scheduled job 0x3ffc3b88, cb 0x400d1c00 at 956540
956540: Running job 0x3ffc3b88, cb 0x400d1c00, deadline 956540
956666: RXMODE_SINGLE, freq=869525000, SF=9, BW=125, CR=4/5, IH=0
957966: irq: dio: 0x1 flags: 0x80
957975: Scheduled job 0x3ffc3b88, cb 0x400d3400 ASAP
957980: Running job 0x3ffc3b88, cb 0x400d3400, deadline 0
958078: EV_TXCOMPLETE (includes waiting for RX windows)
958387: Scheduled job 0x3ffc24bc, cb 0x400d0cfc at 1583386
958712: engineUpdate, opmode=0x900
1583386: Running job 0x3ffc24bc, cb 0x400d0cfc, deadline 1583386
celciusInt2345
buffer[0]=9
buffer[1]=29
sizeof(buffer)=2
1583398: engineUpdate, opmode=0x908
1583583: Uplink data pending
1583746: Considering band 0, which is available at 6589
1584055: Considering band 3, which is available at 0
1584348: No channel found in band 3
1584549: Considering band 0, which is available at 6589
1584858: No channel found in band 0
1585059: Considering band 1, which is available at 1114034
1585384: Considering band 2, which is available at 6589
1585693: No channel found in band 2
1585894: Considering band 1, which is available at 1114034
1586220: Airtime available at 1114034 (channel duty limit)
1586545: Airtime available at 1162873 (global duty limit)
1586865: Ready for uplink
1587027: Updating info for TX at 1583583, airtime will be 2896. Setting available time for band 1 to 1873183
1587628: TXMODE, freq=868500000, len=15, SF=7, BW=125, CR=4/5, IH=0
Packet queued

Code - now reduced to the important only :

#include <Adafruit_GPS.h>
#include <HardwareSerial.h>
#include <lmic.h>
#include <hal/hal.h>
#include "SSD1306.h"

// LoRaWAN NwkSKey, network session key
static const PROGMEM u1_t NWKSKEY[16] = { ... };

// LoRaWAN AppSKey, application session key
static const u1_t PROGMEM APPSKEY[16] = { ... };

// LoRaWAN end-device address (DevAddr)
static const u4_t DEVADDR = { ... };

// These callbacks are only used in over-the-air activation, so they are
// left empty here (we cannot leave them out completely unless
// DISABLE_JOIN is set in config.h, otherwise the linker will complain).
void os_getArtEui (u1_t* buf) { }
void os_getDevEui (u1_t* buf) { }
void os_getDevKey (u1_t* buf) { }

HardwareSerial GPSSerial = Serial1;

// Connect to the GPS on the hardware port
Adafruit_GPS GPS(&GPSSerial);

// Set GPSECHO to 'false' to turn off echoing the GPS data to the Serial console
// Set to 'true' if you want to debug and listen to the raw GPS sentences
#define GPSECHO false

uint32_t timer = millis();

static osjob_t sendjob;

// Schedule data trasmission in every this many seconds 
// (might become longer due to duty cycle limitations).
// we set 10 seconds interval
const unsigned TX_INTERVAL = 10;
// Fair Use policy of TTN requires update interval of at least several min. 
// We set update interval here of 1 min for testing

// Pin mapping according to Cytron LoRa Shield RFM
const lmic_pinmap lmic_pins = {
  .nss = 18,
  .rxtx = LMIC_UNUSED_PIN,
  .rst = 14,
  .dio = {26, 33, 32},
};

SSD1306  display(0x3c, 4, 15);

float temp = 23.45; // Something like 0x41bb999a in IEEE 754 floating-point
int16_t celciusInt = round( temp * 100 ); // convert to signed 16 bits integer: 0x0929
uint8_t buffer[2]; // reserve 2 bytes in memory

void onEvent (ev_t ev) 
{
  Serial.print(os_getTime());
  Serial.print(": ");
  switch(ev) 
  {
    case EV_TXCOMPLETE:
      Serial.printf("EV_TXCOMPLETE (includes waiting for RX windows)\r\n");
      // Schedule next transmission
      os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
      break;  
    case EV_RXCOMPLETE:
      if (LMIC.dataLen)
      {
        Serial.printf("Received %d bytes\n", LMIC.dataLen);
      }
      break;
    default:
      Serial.printf("Unknown event\r\n");
      break;
  }
}

void do_send(osjob_t* j)
{
  // Check if there is not a current TX/RX job running
  if (LMIC.opmode & OP_TXRXPEND) 
  {
    Serial.printf("OP_TXRXPEND, not sending\r\n");
  } 
  else
  if (!(LMIC.opmode & OP_TXRXPEND)) 
  {
    // Clear buffer
    buffer[0] = (char)0;
    buffer[1] = (char)0;

    // Handle high byte (MSB) first; 0x09 for 23.45
    // 0x0929 >> 8 shifts the 0x29 out of memory, leaving 0x0009
    // 0x0009 does not fit in a single byte, so only 0x09 is stored in buffer[0]:
    buffer[0] = celciusInt >> 8;

    // Handle low byte (LSB) next; 0x29 for 23.45
    // 0x0929 does not fit in a single byte, so only 0x29 is stored in buffer[1]:
    buffer[1] = celciusInt;

    buffer[0] = (char) 9;
    buffer[1] = (char) 29;

    Serial.print("celciusInt"); Serial.println(celciusInt);
    Serial.print("buffer[0]="); Serial.println(buffer[0]);
    Serial.print("buffer[1]="); Serial.println(buffer[1]);
    Serial.print("sizeof(buffer)="); Serial.println(sizeof(buffer));

    // Prepare upstream data transmission at the next possible time.
    //LMIC_setTxData2(1, lpp.getBuffer(), lpp.getSize(), 0);

    // Send on port 1, without asking for confirmation:
    LMIC_setTxData2(1, buffer, sizeof(buffer), 0); // 0x0929 for 23.45

    Serial.printf("Packet queued\r\n");
  }
  // Next TX is scheduled after TX_COMPLETE event.

  Serial.println("----");
}

void setup() 
{
  Serial.begin(115200);
  Serial.printf("Starting...\r\n");

  pinMode(16,OUTPUT);
  digitalWrite(16, LOW); // set GPIO16 low to reset OLED
  delay(50);
  digitalWrite(16, HIGH);
  display.init();
  display.setFont(ArialMT_Plain_10);

  // LMIC init
  os_init();

  // Reset the MAC state. Session and pending data transfers will be discarded.
  LMIC_reset();

  // Set static session parameters. Instead of dynamically establishing a session
  // by joining the network, precomputed session parameters are be provided.
  uint8_t appskey[sizeof(APPSKEY)];
  uint8_t nwkskey[sizeof(NWKSKEY)];
  memcpy_P(appskey, APPSKEY, sizeof(APPSKEY));
  memcpy_P(nwkskey, NWKSKEY, sizeof(NWKSKEY));
  LMIC_setSession (0x1, DEVADDR, nwkskey, appskey);

  // Disable link check validation
  LMIC_setLinkCheckMode(0);

  // TTN uses SF9 for its RX2 window.
  LMIC.dn2Dr = DR_SF9;

  // Set data rate and transmit power for uplink (note: txpow seems to be ignored by the library)
  LMIC_setDrTxpow(DR_SF7,14);

  Serial.printf("LMIC setup done!\r\n");

  // Start job
  do_send(&sendjob);

  Serial.println("----");

  delay(1000);

  // Ask for firmware version
  //GPSSerial.println(PMTK_Q_RELEASE);
}

void loop() 
{
  // Make sure LMIC is ran too
  os_runloop_once();
}

[itofficeeu]

I have further information.

I reset the program to use Cayenne.lpp AND use up to 51 bytes. - Like this:

include

CayenneLPP lpp(51);

` lpp.reset(); lpp.addTemperature(1, 23.45);

LMIC_setTxData2(1, lpp.getBuffer(), lpp.getSize(), 0); // 0x0929 for 23.45

And of course change the TTN to handle the payload as Cayenne data.

`

It now spits out sort of random data like before, even if I always just tries to send the same value '23.45'. But by accident does it produce usable data. Not the data I send, but still some sort of 'usable' data.

The 'usable' data are these

Payload: 2C 65 EA 98

Fields: { "luminosity_44": 60056 }

This should actually be 23.45 what I was sending. But if I look the this link: https://mydevices.com/cayenne/docs/lora/#lora-cayenne-low-power-payload

I can convert the EA 98 to 60056,

And the table over Data Types tells me that '65' in front of the value stands for Illuminance Sensor - In my case named 'luminosity_44* by something I don't control.

I do not yet know what 2C stands for. - maybe the channel?...

Another 'usable' data was:

Payload: 32 03 92 98

Which again according the Cayenne data types can be transferred to: { "analog_out_50": -280.08 }

[itofficeeu]

More information;

I have now tried to make a buffer like a Cayenne format - like this:

` uint8_t buffer[4]; // reserve 4 bytes in memory

buffer[0] = 0x00; // - I don't know hat to put.
buffer[1] = 0x67; // For the data type Temperature Sensor
buffer[2] = 0x09; // First part of the float (MSB)
buffer[3] = 0x29; // Second part of the float (LSB)

Serial.print("buffer[0]="); Serial.println(buffer[2]);
Serial.print("buffer[1]="); Serial.println(buffer[3]);
Serial.print("sizeof(buffer)="); Serial.println(sizeof(buffer));

// Send on port 1, without asking for confirmation:
LMIC_setTxData2(1, buffer, sizeof(buffer), 0); // 0x0929 for 23.45

`

But this does not lead to any usable or 'usable' data. Just randon like the first time.

[matthijskooijman]

I had a look over your code and output, but could not find anything obviously wrong. I'm a bit at a loss here, unfortunately.

Did you try the ABP example sketch as-is? Does that work?

[itofficeeu]

Yes, that works.

For ESP32 / Heltec WiFi LoRa 32 must the pin map be changed to

const lmic_pinmap lmic_pins = { .nss = 18, .rxtx = LMIC_UNUSED_PIN, .rst = 14, .dio = {26, 33, 32}, };

I could send the message "Hello World!" as well as the temperature data as Cayenne:

lpp.reset(); lpp.addTemperature(1, 23.45); LMIC_setTxData2(1, lpp.getBuffer(), lpp.getSize(), 0); // 0x0929 for 23.45

I must simply have screwed up the other code during working with it. I will try to see if I can find the error.

[matthijskooijman]

Good to hear there's some progress there :-)

If you find the problem, care to report here? I'm curious what could cause this erratic behaviour...

[itofficeeu]

Hunted down to the APPSKEY having just one single letter/number wrong.

Unfortunately is the font used in Arduino IDE not clear in the difference between 8 and B. :(

• This can only be lead back to somebody loving a "good" looking font more, than something that is readable. :(

This has "only" cost me 3 days. At least it works. Time to move away from Arduino IDE. What a crap.

[gizmocuz]

@itofficeeu , plenty of choices ;) https://www.intorobotics.com/alternatives-standard-arduino-ide-one-choose/

[itofficeeu]

@gizmocuz - Wow! Thanks! Nice! :D Wonderful link! :D & thanks to @matthijskooijman too for being here! :D