New LMIC can be busy after TXCOMPLETE: need to document, fix sample apps for clarity

[terrillmoore]

The behavior of the LMIC changed between 2.3.2 and 3.10.0, which requires that apps be more careful about sending messages. With the upgraded MAC code, the LMIC and the network can get involved in lengthy discussions after an uplink. However, the LMIC correctly notifies the app whenever a transmission completes. We need to:

• [ ] document the behavior clearly
• [ ] add information about how best to deal with the situation of extended network conversations
• [ ] most important, update the sample apps to follow best practices.

Original comment from @DeveloppeurPascal follows.

I found the reason of the problem : automatic MAC response throw TXCOMPLETE event, so I send next message, but the previous payload was pending... a new payload is sent during send process from the previous one, so it disappear.

We must modify the do_send() function like this :

  if (LMIC.opmode & OP_TXRXPEND) {
   Serial.println(F("OP_TXRXPEND, not sending"));
  } else if (LMIC.opmode & OP_TXDATA) {
   Serial.println(F("OP_TXDATA, not sending"));
  } else {
      LMIC_setTxData2(1, LoRaWanMsgToSend, LoRaWanSizeOfMsgToSend, 0);
      // Next TX is scheduled after TX_COMPLETE event.
  }

Please add the OP_TXDATA test in your samples to prevent the problems I add.

Originally posted by @DeveloppeurPascal in https://github.com/mcci-catena/arduino-lmic/issues/545#issuecomment-594824824

[cyberman54]

@terrillmoore a question in this context: i added code to my app to print uplink and downlink MAC commands. It's not fully working yet, i am not sure if i grab all MAC commands.

With LMIC 3.1.0, what would be suitable place to process MAC commands by an app?

I currently do it in the lmic_rxmessage_cb_t RX callback function. I see some downlink MACs, but seem to grab to much data from LMIC.frame whch cannot be resolved in MAC CIDs. This way i see also uplink MACs, but not the first which transmit immediately after join (in my case this is a DevTimReq).

[Rock-N-RollaMann]

First just wanted to say thank you to everyone who contributes(d) to this library. Moreover just wanted to add my experience I had this problem with the update version of the "Lmic" library and added the OP_TXDATA to my code, however I had the same results. Currently I am using the older version of the "Lmic" and everything is working fine.

[altishchenko]

There is a subtle logic issue here. If for some reason the network server does not acknowledge a confirmed uplink, but still sends some commands with port 0, fSuccess flag in the call to txMessageCb will still be set true, because pendTxConf flag gets reset to 0 in the 'open code the logic' block ( lmic.c:2351 ) and is further treated as unconfirmed uplink by TXCOMPLETE event. Because of this code block, pendTxConf is an unreliable source of confirmed/unconfirmed indication and can't be used in the event logic processing functions.

[pomplesiegel]

I believe this is related:

I am seeing behavioral changes in v3.x (vs 2.x) such that

1. The program used to transmit a full payload right after a join, but this no longer happens
2. The second message has no payload, and this happens occasionally after many transmits as well

These appear on TTN like so:

[Rock-N-RollaMann]

Yea, I agree I've been seeing that behavior as well with my node (feather m0). However, it sends "payload [not provided] once" and then goes back to normal without downlinks confirmations. Which you would see in the older "lmic" libraries. Moreover, I think it is okay, and it will probably help you save battery life with fewer downlinks.

[altishchenko]

It is ADR and other device control stuff like DevStatusReq/Ans etc. CFList and other settings the server pushes down to the device, queries it about battery and signal quality. Current LMIC does not wait for the application to provide some data frame for the reply piggybacking, it just sends the reply asap in an unconfirmed frame, giving you EV_TXCOMPLETE event as a result. Once, twice or many times you see this depends on your network server, gateway and frequency plan configuration on the device, as well as on signal quality.

[pomplesiegel]

@altishchenko thank you for the context! Do you know an easy way to have the device still send its original intended payload, as right now it's totally unwittingly skipping a sampling interval directly following a downlink. I would rather not have to turn off ADR and link-check to fix this. Thank you!

[altishchenko]

@pomplesiegel Michael, not for the unconfirmed uplink. If your device sends confirmed uplinks, then you could first check for ACK in EV_TXCOMPLETE and also later check the SUCCESS return from the LMIC_setTxData2 (or more hacky for OP_TXDATA) as a signal that there was no extra data queued. Everything is much more complicated for the unconfirmed data - you can't distinguish between your transmissions and library's originated ones. As I said earlier, no mechanism is provided to piggyback control data on application provided data frames. As an another measure, library could use yet another special flag to mark its own transmissions and bypass EV_TXCOMPLETE generation in such case. Terry? (@terrillmoore)

[pomplesiegel]

Hmm indeed I don't do confirmed uplinks, as they are so expensive. The thing I don't understand is why the behavior has changed between LMIC 2.x and 3.x, leading to these empty payloads? That's the only thing that I mind, as if we are sampling infrequently, that could potentially be a lost hour or day of sampled data. I believe that did not used to happen.

[altishchenko]

@pomplesiegel Michael, then the only way out is attempting to send your data and checking the return status of the LMIC_setTxData... If it returns LMIC_ERROR_BUSY you should retry your transmission in a couple of seconds (and keep retrying until it returns SUCCESS), that's it.

[pomplesiegel]

@altishchenko, thank you for the help! Would that mean if I directly scheduled another transmit after "setTx" returns non-zero, that would take care of the issue? Like below?

        if ( LMIC_setTxData2(1, UplinkTransmitPayload, UPLINK_PAYLOAD_SIZE, 0) )
        {
            Print("ERROR while uploading! 'LMIC_setTxData2' returned non-zero. Trying again");
            os_setTimedCallback(&sendjob, os_getTime() + sec2osticks(1), do_send);
        }

[pomplesiegel]

It seems my solution above didn't fix the issue, at least the uplink payload after the first downlink, which was again empty:

OP_TXRXPEND, not sending
.
1130474
: 
EV_TXCOMPLETE (includes waiting for RX windows)

so LMIC_setTxData2() still returned 0

[altishchenko]

@pomplesiegel Oh no! You won't get rid of that empty payload message. No. You can filter it out on the application side by the port being 0 though. The approach you have above is correct, given you schedule transmissions upon receipt of the EV_TXCOMPLETE message. But care should be taken here. Consider the scenario:

1. You send data
2. You receive EV_TXCOMPLETE and schedule another send.
3. Send fails, because the transmit buffer is occupied by the protocol reply on port 0.
4. You get one more EV_TXCOMPLETE saying that the transmit buffer is empty now.

There is only one probable way to distinguish between the two events: checking OP_TXDATA flag upon receipt of the event in your events handler. If it is set, don't schedule another transmission, wait for a guaranteed second EV_TXCOMPLETE.

I must also point out this (and somewhat contradict my earlier words): The reason you are getting the first event, is because you may have data received during your RX window, even if it is control data. More so, you can inspect it and override the replies scheduled for transmission by the library :). The second EV_TXCOMPLETE - it seems there is no reason for it to be passed down to the App, that it should be hidden, but no again - you may have data received again via your RX window and if this event will be squashed, you will lose that data. Surely, the check in the library can be made for data presence, but making choice on this will make things even worse and more complicated.

[terrillmoore]

Sad to say, it's all true. Correcting things in the air interface exposes the app to a number of complications. You must now either check that the LMIC is not busy before transmitting, or check the return status, looking for the busy error. Bear in mind that the network can keep the LMIC busy for an arbitrary period of time -- everytime the LMIC sends an uplink, the network can send a MAC or application message during the RX window, and that MAC message can require a reply... and when the LMIC transmits the reply, there's another downlink window. This can repeat indefinitely. So your app needs a mechanism to deal with this (called in the hardware biz an "elasticity buffer", in the software biz, a "queue"). This reduces the simplicity a lot.

You might want to look at the code in https://github.com/mcci-catena/arduino-lorawan/, which basically gives you a classic callback-driven API to the low-level LMIC APIs. I did not want to totally rip out the old API, but perhaps I should have.

In the next release (if I ever get some time) I think I should look into allowing you to configure the LMIC to delay network uplinks until the next application uplink. That would make the application behavior much simpler, at the cost of delaying gratification to the network when it wants an answer from the device on port 0 (for ADR operations).

[terrillmoore]

The thing I don't understand is why the behavior has changed between LMIC 2.x and 3.x, leading to these empty payloads?

As to why the behavior changed: network MAC downlinks fundamentally didn't work in 2.3.2. This meant that most downlinks were dropped, so there was much less of this kind of behavior.

[altishchenko]

My extra 5 cents on the whole thing: You should always have a mechanism to delay and re-send your data no matter what is the library implementation. If you are sending unconfirmed frames - get ready to lose data, if you are sending confirmed frames - get ready to resend them. The network may go down, the air may get polluted, the server may reboot - anything - you lose data. If you are careful and data is precious - you will get a NACK or another error telling you that you should try again later, may be after re-join even, so you should have an external mechanism or queue to store your data until it is successfully transmitted and that transmission is confirmed by the server or application behind it. The library will not do this for you, the library implements the protocol which has just this many retransmission attempts after which it will report to you that it couldn't do anything and you should try and solve the problem on the level higher. Sorry, got carried away. It is a context.

What I was trying to say more plainly - you must check the return code from the LMIC_setTxData and really act upon it. Everything else is a hack. If it returns the BUSY error, which is normal, you should save your data and retry later or discard it and retry later with the new data. Terry is right about conversations. You should see them happening when using ABP mode and Actility ThingPark - every extra channel apart from the default ones gets configured by one server-to-device message (with reply of course) and later gets switched on by another message (with reply again). Now let's count: EU = 8 channels typical (16 max), 3 of them default, leaves 5 channels to configure and enable. With the best effort approach that will result in 3-4 downlink messages from the server and 4 uplink replies from the device. Just to setup channels. Not even starting the conversations on signal quality, 'recommended' spread factor/data rate, transmission power and operating frequency for the moment 'now'.

[pomplesiegel]

Enlightening discussion! Thank you both very much.

I have since refactored and now will retry (until a defined # of tries, after which we go sleep and try again upon waking up) to transmit our intended data payload, setting and checking a flag based on the logic described at the top of this thread by @terrillmoore:

if (LMIC.opmode & OP_TXRXPEND) {
   Serial.println(F("OP_TXRXPEND, not sending"));
  } else if (LMIC.opmode & OP_TXDATA) {
   Serial.println(F("OP_TXDATA, not sending"));
  } else {
      LMIC_setTxData2(1, LoRaWanMsgToSend, LoRaWanSizeOfMsgToSend, 0);
      // Next TX is scheduled after TX_COMPLETE event.
  }