Closed czuvich closed 4 years ago
@czuvich With TTN (on EU868?) you may try a LMIC_setLinkCheckMode(true)
before join.
I don’t think Join applies since I’m using ABP, so setting the linkCheck flag won’t really help. I’m on US915
Sorry you're having problems.
If OTAA join doesn't work, high confidence: downlink doesn't work at all. (And if downlink isn't working, timing is broken on your platform; could be another variant of the interrupt problems.)
In your reported problem, TTN US is trying to control your state by sending the downlink. I hypothesize that you never receive and ack the message (because downlink is not working), so on every uplink, TTN US repeats the message.
I have not recently tested LMIC_setAdrMode(0)
. Unfortunately, I don't have the resources to test everything; and I never use LMIC_setAdrMode(0)
(or ABP) in anything I do: ABP is really a last resort.
It would really help me out if you could use the git bisect
procedure from #541, which I repeat below.
git bisect
to find the commit that broke your codeIf you're not familiar with using git bisect
, it's straightforward:
git clone
to checkout arduino-lmic from githubcd
to the arduino-lmic directory.git checkout v3.1.0
git bisect start
git bisect bad
(because v3.1.0 is bad)git bisect good v3.99.3
(because v3.99.3 is ok)git bisect good
; otherwise git bisect bad
.git log --oneline
to show the commit history from the bad commit down, and post that here.Thanks for the reply! Responses below.
If OTAA join doesn't work, high confidence: downlink doesn't work at all.
Downlink works fine using ABP regardless of which version of the library (between v3.0.99 or 3.1). I've verified this by scheduling downlinks to the node, and I am seeing the payload on the node.
Things break when I turn off ADR on v.3.1.0. I'll take some time this week to run git bisect on this particular issue and post my findings here since I can consistently reproduce. I really appreciate the help!
My hunch is that the findings in #524 are directly related to the OTAA issues me and others are having.
In all honesty, I'm personally not a huge fan of OTAA since I "join" via my backend (not directly from a node - my node is registered through my mobile app, which then auth's to TTN). I personally think it overcomplicates the end node, but I'm going to post this on TTN and see some responses. I would say 70% of issues with LoRaWAN stacks are related to OTAA.
If using a completely private network, ABP is ok; but on a standards-based network, it is very hard with cheap hardware to achieve the strict requirements of the spec related to reuse of FCntUp.
Downlink is the issue with OTAA; it must work in order for OTAA to work. But on the other hand, generally if a device joins, the remaining downlink features are less challenging. So making OTAA work is well worth the effort.
In any case, #524 is almost certainly related to your problems with downlink. I am working on a portable fix; there's no reason for the underlying problem other than ... that's the way they did it the first time.
However, with all the changes in the MAC layer, it's very possible thatLMIC_setAdrMode(0)
was inadvertently disabled. Luckily the commits are mostly fine-grained enough that I suspect it will be obvious, once we find where it broke. If so, we need to fix it, independently of the downlink problems on your platform.
Thank you for the very detailed explanation! I am pretty new to LoRaWAN, so I have a lot to learn. Pros/Cons of OTAA and ABP seem to make sense. I'm thinking one day, the software stacks for LoRaWAN will be as ubiquitous as HTTP/HTTPS in such a way that the application developer will not be concerned with either. It'll be the backend + library that negotiates the authentication. Anyways, I ran git bisect. I am new to doing this; however, I think I did it. Here are my logs. Let me know if this is what you're looking for:
ff21bc7ef7e369bf071e00294a0d6a2ccb28dbb2 is the first bad commit
MacBook-Pro-2576:arduino-lmic Clay$ git log --oneline
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
bec1cfd Fix #383: clean up compile warnings found by CI testing
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
bec1cfd Fix #383: clean up compile warnings found by CI testing
3206214 Don't build EU compliance sketch for now -- too big
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
bec1cfd Fix #383: clean up compile warnings found by CI testing
3206214 Don't build EU compliance sketch for now -- too big
eee3933 Shink compliance a little
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
bec1cfd Fix #383: clean up compile warnings found by CI testing
3206214 Don't build EU compliance sketch for now -- too big
eee3933 Shink compliance a little
b57531e Improve clock cal for STM32L0 in compliance sketch
:...skipping...
8a62487 (HEAD, refs/bisect/good-8a62487ec9bae478843a6302c8e2efa1c30dce95) Merge pull request #497 from mcci-catena/issue495
897d5d5 (refs/bisect/good-897d5d5815a1ccc39162aa6dfce42dd95f1b0982) Fix #487: add new event codes to examples
19057c4 Fix #495: improve display of AppSKey and NwkSKey
e961979 Merge pull request #490 from mcci-catena/issue489
04ce480 (origin/issue489) Fix #489: BSPs now understand au915 not au921
5322dd1 Merge pull request #472 from manuelbl/invalid_len_comparison
6225a56 unsigned type for length
98edf2d Merge pull request #465 from frazar/bump-arduino-version
d603e2e Fix comparison for max data frame length
6eeaf6d Bump Arduino IDE version to 1.8.10
852f348 Merge pull request #452 from mcci-catena/issue447
8810228 Fix #451: improve documentation on region selection
af0f23b Clean up warnings in compliance sketch, more AVR-isms
467d137 Clean up warnings in compliance sketch, more AVR-isms
ea09b2d Version is v3.0.99.5
f8f9bbc Fix #450 Fix #445: LMIC_clrTxData more carefully
0c06144 Fix #448: typo in AU915 code
b1d79b5 Fix #446: AS923 don't report join failure till all ch scanned
ccfcc75 Version is v3.0.99.4
3209e33 Fix #448: AS923, AU915 enforce dwell time based on TxParam
b7d4433 Fix #447: Default AS/AU UplinkDwellTime must be 1
3364725 (tag: v3.0.99.3) Merge pull request #443 from mcci-catena/issue423
e691f50 Update documentation / revision history
bec1cfd Fix #383: clean up compile warnings found by CI testing
3206214 Don't build EU compliance sketch for now -- too big
eee3933 Shink compliance a little
b57531e Improve clock cal for STM32L0 in compliance sketch
8df712a version is v3.0.99.3
I was browsing through that commit ff21bc7ef7e369bf071e00294a0d6a2ccb28dbb2, and that's a huge commit.
@czuvich in a big LORAWAN network the network server should be able to influence the radio parameters of the nodes. This is the purpose of MAC commands in LORAWAN. They can only work if downlink is working. Thus, we need downlinks in LORAWAN networks. And if we have them, we can use OTAA.
I am running MCCI LMIC 3.1.0 on different nodes, mostly ESP32 with multitasking apps. No problems with crashes so far, and i don't see the "downlink after every uplink" issue with TTN any more (had this some time ago, but it disappeared, don't know why).
@czuvich in a big LORAWAN network the network server should be able to influence the radio parameters of the nodes. This is the purpose of MAC commands in LORAWAN. They can only work if downlink is working. Thus, we need downlinks in LORAWAN networks. And if we have them, we can use OTAA.
Yup makes sense. And I'm guessing as the LoRaWAN spec evolves, so will the software stacks, and soon the app developers won't have to concern themselves with OTAA vs ABP (at least on the end node side).
I am running MCCI LMIC 3.1.0 on different nodes, mostly ESP32 with multitasking apps. No problems
Good to know it's working on ESP32. There are others running into the same issues as myself with different hardware setups. Hopefully, this latest git bisect sheds some light on it.
It is a merge commit. You need to go through the branch. It appears that Git doesn’t walk you through the branch directly, which is unfortunate.
I suggest git checkout -b tmm-issue453 3b9b02d
; and then start the git bisect again (with the branch checked out). Use the first good commit as 05d304f or e961979.
You will see that there is are points where the compile fails. You’ll want to do git bisect skip
to get past them, if run into one.
--Terry
Ok.. ran it on the branch. Here are my results. Again thanks for your help!
git checkout -b tmm-issue453 3b9b02d
306330fcb5be10c9897f96d2885af530d168a5d3 is the first bad commit
commit 306330fcb5be10c9897f96d2885af530d168a5d3
Author: Terry Moore <tmm@mcci.com>
Date: Mon Dec 30 11:19:58 2019 -0500
Fix #508: don't depend on delayMicroseconds()
:040000 040000 86be995293f7d8b51b246500ee52063aada0ac2e fddb52b5f498cdf9185610fcaebc65f34206cbf7 M src
The --oneline log is really long, so I truncated it off. Let me know if you need all of it.
MacBook-Pro-2576:arduino-lmic Clay$ git log --oneline
306330f (HEAD, refs/bisect/bad) Fix #508: don't depend on delayMicroseconds()
4a267e5 (refs/bisect/good-4a267e598409ed31a947503f1ec8b3f55c90f31c) Fix #504: AVR 32-bit shifts need casts on left operand
a4044c0 Partial Fix #442: clean up window calculations based on lab study
9671dea Part of fix #442: Detect late RX/TX launches; improve HAL timing a bit
e59c123 Version is v3.0.99.9
5933c64 (refs/bisect/good-5933c6415046f12bb6fa0a34249c49b6e9995e8a) Fix #493: further correct RSSI calcs (sign error, fix non-linearity correction)
f213c43 Fix #502: add os_radio(RADIO_TX_AT), version is 3.0.99.8
b912a44 Change type of rxsyms to rxsyms_t
044807a Since there are multiple rxmodes, log the value
4bae5fa Fix #467: make rxsyms a 2-byte value
b7b640d Add logs with flags so we can diagnose timing probs
01ef67b Get rid of another ASSERT() in FSK path
d18c906 Bump version to v3.0.99.7
676dab8 For issue #467, assume 10-bit rxsyms (partial correction)
1dcfbfa Update RadioDriver.md docs
93539d9 (tag: V3.0.99.6) Fix #493: correct RSSI calcs; v3.0.99.6
f428f11 Add documentation on radio driver
43b79c0 Improve documentation of events
f4b2393 Fix Heltec v2 pinmap and add more Heltec pinmaps
b86692b Log rxsyms in compliance app
540034c Fix #466: recognize TTGO and Heltec boards
:...skipping...
:
I'm guessing the issue is in this line right here.. And I'm wondering if this is the same line that is causing other issues in downlinks/RX.
while (delta_time(time) > 0)
Interesting.. so I fixed my problem with disabling ADR using the latest version of the library (v3.1); however, I do still see an uplink for every downlink with ADR turned back on. Here's what I did in the library:
// deal with boards that are stressed by no-interrupt delays #529, etc.
//#if defined(ARDUINO_DISCO_L072CZ_LRWAN1)
# define HAL_WAITUNTIL_DOWNCOUNT_MS 16 // on this board, 16 ms works better
# define HAL_WAITUNTIL_DOWNCOUNT_THRESH ms2osticks(16) // as does this threashold.
//#else
//# define HAL_WAITUNTIL_DOWNCOUNT_MS 8 // on most boards, delay for 8 ms
//# define HAL_WAITUNTIL_DOWNCOUNT_THRESH ms2osticks(9) // but try to leave a little slack for final timing.
//#endif
I used the timing defined for the DISCO board, and no more crash. I was seeing the app lock up in the hal_waitUntil method. Could this also be related to other issues with OTAA I'm seeing as well? I haven't tried it.
Any suggestions or feedback? I’d be willing to send actual hardware for troubleshooting.
It feels like the issue is directly related to the timing of downlinks and the hard coded delay timing vars. Perhaps if you can shed some light on why it’s done that way I can help.
Sorry, a little distracted here. All of these issues are fundamentally related to #524. As soon as I have finished a few things that have to be done for MCCI's main business, I plan to fix #524. If you can wait a few days, my guess is that this will fix the problems.
Sounds great! Again thanks for the outstanding work! And let me know if there’s anything I can do on my end.
I hope you and your family are ok. I assume there won't be too much progress on the timing in #524 during this time correct?
Hi @czuvich, we're all fine, hope the same for you. I run a company, which has been keeping me busy. I've committed a fix for #524, and am about to post an update. I believe this will improve things a lot.
@terrillmoore Glad to hear your family is doing ok! I downloaded your latest update and it looks like OTAA is working for my board along with downlinks. Thank you so much for the great work! Now to convert my backend registration to OTAA :D
I am using an nRF52840 Express from Adafruit and a HopeRF95. I have been having various issues with the latest version of the library (v3.1), and I downgraded to v3.0.99 which has resolved a few problems; however, I am noticing the following.
Library version: v3.1.0 Code: ttn-abp example
I am using ABP (OTAA just doesn't work for my hardware), and I was seeing a downlink for every uplink to the TTN Gateway on my End Node. I looked at the traffic, and TTN is sending a LinkADRReq for every uplink. I don't think that's expected behavior. I assumed LMIC would update the ADR info and send an ACK back; however, I don't think that's happening.
I then tried the LMIC_setAdrMode(0) in the setup(), and my end node froze after sending the uplink. I saw the uplink in the gateway.
Library Version: v3.0.99
Followed the same steps as above, but everything worked. The app did not crash, and I am no longer seeing a downlink for every uplink.
I do think there's an overall issue with timing and the latest update to the repo (v3.1.0).
Having said that, I was browsing the repository issues, and it'd be awesome to include some updated documentation on a more "best practices" workflow of using this fantastic library (for instance, I'll probably stitch together 552 with a few others to get this working for my app. GREAT job on this library, as I'm sure it requires a lot of time and testing.