ESP32 Deep Sleep support without ttn.join() after every wakeup

[Bravd42]

Hello,

I'd like to send my Heltec Wireless Stick into deep sleep after OTAA without joining on each wakeup. As far as I understand I have to preserve some counters and the SessionKeys.

I saw that saveKeys(); gets me half the way by storing all OTAA parts that should not change anyway to flash. But what about the session state Frequencies, counters, … and what not.

I'd like to preserve them in RTC_RAM. Is this possible without modifiing your library?

Regards, Bravd

[enwi]

I was just working on that today, and have a working example, give me a second. Also automatic session storing/restoring is implemented once #7 gets merged

[enwi]

Example added with bff82ea91707bf60c31d2541f78f2f5dd6847218

[Bravd42]

Hi enwi, thanks for the reply.

I examined your code and came to the conclusion that this is exactly what I try to avoid doing. In this example the node has to call ttn.join() every time he wakes up which is very much frowned upon by The Thinks Network, because it is adding unnecessary traffic on gateways: https://www.thethingsnetwork.org/docs/devices/bestpractices.html#otaa-best-practices

What I'd like to do have is some code which preserves all LoRa related variables in RTC_Memory so I only have to call ttn.join() at first boot up.

Regard, Bravd

[enwi]

@Bravd42 It does not rejoin when you call join(). If you look at the implementation you can see that in the case where there is a stored session the joining procedure is skipped:

bool TTN_esp32::join()
{
    bool success = false;

    if (!provisioned && !session)
    {
        restoreKeys();
    }

    if (session)
    {
        Serial.println("Using stored session to join");
        devaddr_t dev_addr = dev_adr[0] << 24 | dev_adr[1] << 16 | dev_adr[2] << 8 | dev_adr[3];
        personalize(0x13, dev_addr, net_session_key, app_session_key);
        success = true;
    }
    else if (provisioned)
    {
        Serial.println("Using stored keys to join");
        LMIC_setClockError(MAX_CLOCK_ERROR * 7 / 100);
        LMIC_unjoin();
        LMIC_startJoining();

#ifdef DEBUG
        Serial.println("joined");
#endif // DEBUG

        xTaskCreatePinnedToCore(loopStack, "ttnTask", 2048, (void*)1, (5 | portPRIVILEGE_BIT), TTN_task_Handle, 1);
        success = true;
    }
    else
    {
        ESP_LOGW(TAG, "Device EUI, App EUI and/or App key have not been provided");
        Serial.println("No keys provided");
    }
    return success;
}

Also #7 is there because of this https://www.thethingsnetwork.org/docs/devices/bestpractices.html#otaa-best-practices

[Bravd42]

English below.

Hallo enwi,

danke dass ihr euch so gut um mich kümmert.

Ich hatte gestern Nacht nur den Beispielcode gelesen und nicht den Library code von ttn.join(). Heute Abend hab ich vor den Beispielcode zu testen. Wenn das funktioniert kann ich den LoRa-Part in meinem Projekt endlich umsetzen. Dann brauch ich nur noch Empfang auf meiner Bienenwiese.

Gruß, Bravd

---

Hi enwi,

sorry, I wasn't aware of that. I'm going to test your code later today. If this is working like intendet the LoRa related part of my project is finally solved. All I need from now on is LoRa connectivity at the meadow where my bees are located.

Thank you for your kind support.

Regards, Bravd

[enwi]

Aber nicht vergessen die Branch von meinem Fork zu nehmen But don't forget to use the branch of my fork

[Bravd42]

Hi,

danke, das hätt ich vergessen.

git clone https://github.com/enwi/TheThingsNetwork_esp32.git git checkout store-session-and-refactoring

correct?

Gruß Matthias

[enwi]

Yes that's correct

[Bravd42]

I noticed that saveKeys() uses NVS to store the session. If I'm not mistaken storeSequenceNumberUp() will be written on flash every time ESP32 goes to sleep which, will wear out the FLASH in no time.

May I suggest storing it in RTC_Memory instead? This part of RAM will retain all Data despite deep sleep.

All you have to do is defining:

RTC_DATA_ATTR uint32_t NVS_FLASH_KEY_SEQ_NUM_UP;

You don't even have to worry about bitshifting.

From that point forward you use it like any other variable but don't have to worry about loosing its values.

For convenience one could avoid NVSHandler all together an store all in RTC:

RTC_DATA_ATTR uint8_t buf_dev_eui[8]; RTC_DATA_ATTR uint8_t buf_app_eui[8]; RTC_DATA_ATTR uint8_t buf_app_key[16];

RTC_DATA_ATTR uint8_t buf_dev_adr[4]; RTC_DATA_ATTR uint8_t buf_net_s_key[16]; RTC_DATA_ATTR uint8_t buf_app_s_key[16]; RTC_DATA_ATTR uint8_tbuf_seq_num[4];`

Better still: store it in NVS if you intend to depower the device, and in all other cases just keep it in RTC_MEMORY.

Regards, Bravd

[enwi]

@Bravd42 Good call, but some deep sleep modes do not preserve the RTC_Memory and a full power down will also loose those values

But also according to this you can expect to only get 10,000 to 100,000 write cycles, so this might need to be adressed and be disucessed in more detail.

[Bravd42]

@enwi as far as I know only ESP32 Hibernation mode and power down looses RTC_Memory. In that case one would use NvsWrite to preserve it.

I my project I'm preserving measurement data in RTC_Memory while using esp_deep_sleep(). All lighter sleep modes preserve it as well.

[enwi]

So could it be possible to detect hibernation and power down?

Also is it possible to store arrays that are not read only?

Unfortunately, any string constants used in this way must be declared as arrays and marked with RTC_RODATA_ATTR, as shown in the example above.

https://docs.espressif.com/projects/esp-idf/en/latest/api-guides/deep-sleep-stub.html#loading-data-into-rtc-memory

[enwi]

Just as an idea, technically we can still store

uint8_t dev_eui[8];
uint8_t app_eui[8];
uint8_t app_key[16];

uint8_t dev_adr[4];
uint8_t net_s_key[16];
uint8_t app_s_key[16];

in the NVS, since they are really only stored once. Then store the sequence number in RTC mem with

RTC_DATA_ATTR uint32_t seq_num;

And only update the NVS sequence number every 100th time (0, 100, 200, 300, ...). Then if we cold start just read NVS sequence number and add 100 to it (since we don't know where we left off, max allowed gap is 16384). Otherwise use the RTC sequence number.

[Bravd42]

Yes and yes. If you power up ESP32 it is aware about how it got there. But even simpler in my code I keep track by a boot counter like in the deep_sleep example code. So if the counter is 0 I know my RTC_Memory was erased. If it > 0 I know I only was asleep. I'm guessing hibernating feels like power down in that sense.

Concerning the array: I'm in fact use an array in RTC Memory to store Data as a ringbuffer

[Bravd42]

I'm all for storing all keys in NVM. Storing the sequence number only every 100 th time won't get us far. If you power down at 1 Message before the next NVM storage cycle, you have to send 99 Messages till gateways accept the 100th Message as the first valid.

It would be best if it was possible to trigger an sequence Number write over the API.

So I could do something like detecting a button push, an trigger:

ttn.storeSeqToNVM(); esp_hibernate();

or savely power off the device.

[enwi]

Technically messages are only not accepted when the sequence number is lower than the one TTN knows, otherwise it should accept the new/higher count number or once you could not send one message you can't send any more messages

[Bravd42]

I didn't get the plus 100. Now i get it.

[Bravd42]

Every 100 Messages sounds good. I'm planing on sending 4 Messages per hour, so it would store the sequence once a day. That's much easier as triggering by API call.

[enwi]

I mean you could also increase the value, but I don't like this solution that much as it is only a workaround of the real problem, which still exists

[Bravd42]

A little off topic. I don't get how often I'm allowed to send per hour. My airtime was 1200ms per Message. Does that mean I'm allowed to send 30 times?

3600.0 / 100 / 1.2 = 30.0

[Bravd42]

I mean you could also increase the value, but I don't like this solution that much as it is only a workaround of the real problem, which still exists

The problem beeing NVM wearout?

[enwi]

The problem beeing NVM wearout?

Yes, because we know it is going to wear out, but we are just delaying the time until it is going to wear out, but not solving the problem

[enwi]

A little off topic. I don't get how often I'm allowed to send per hour. My airtime was 1200ms per Message. Does that mean I'm allowed to send 30 times?

3600.0 / 100 / 1.2 = 30.0

Maybe this helps https://www.thethingsnetwork.org/forum/t/limitations-data-rate-packet-size-30-seconds-uplink-and-10-messages-downlink-per-day-fair-access-policy-guidelines/1300

[Bravd42]

A little off topic. I don't get how often I'm allowed to send per hour. My airtime was 1200ms per Message. Does that mean I'm allowed to send 30 times? 3600.0 / 100 / 1.2 = 30.0

Maybe this helps https://www.thethingsnetwork.org/forum/t/limitations-data-rate-packet-size-30-seconds-uplink-and-10-messages-downlink-per-day-fair-access-policy-guidelines/1300

Thank you that helped me and my NVM wearout ...

[enwi]

@Bravd42 Can you try the new code that I just pushed? I have some issues testing it, since my devices don't receive the join accept.

[Bravd42]

@envi It try it later today.

[Bravd42]

@enwi looks good so far. I added #define DEBUG and my credentials to your example ttn-otaa-sleep. Here is the output:

Starting
Wakeup was not caused by deep sleep
[checkKeys] provisioned no session
Using stored keys to join
joined
65929: EV_JOINING
Joining TTN .............450389: EV_TXSTART
..........771466: EV_JOINED
netid: 19
devaddr: 260122BB
artKey: 
nwkKey: 
joined !
Waiting for pending transactions...  took 0ms
784720: EV_TXSTART
Packet queued
Temp: 53.333332 TTN_CayenneLPP: 1 67 0215
Waiting for pending transactions... 915316: EV_TXCOMPLETE (includes waiting for RX windows)
 took 2100ms
Going to sleep!
Starting
Wakeup caused by touchpad
[checkKeys] provisioned session
Using stored session to join
Joining TTN 
joined !
Waiting for pending transactions...  took 0ms
65954: EV_TXSTART
Packet queued
Temp: 53.333332 TTN_CayenneLPP: 1 67 0215
Waiting for pending transactions... 196628: EV_TXCOMPLETE (includes waiting for RX windows)
 took 2100ms
Going to sleep!

My own gateway receives packets with SF7 BW125. Counter increments as it should.

[enwi]

So this issue can be closed once #7 gets merged

[enwi]

@rgot-org or @Bravd42 You can close this issue now