search

mcci-catena / arduino-lmic

LoraWAN-MAC-in-C library, adapted to run under the Arduino environment
https://forum.mcci.io/c/device-software/arduino-lmic/
MIT License
637 stars 207 forks source link

Compliance Testing #924

Open DylanGWork opened 1 year ago

DylanGWork commented 1 year ago

Please give a clear and concise description of the problem you're facing and what you'd like help with. Hi all,

I am having trouble passing LoRaWAN Compliance testing on V1.0.3 on a Redwoodcomms compliance tester.

It does not pass many of the tests our consultant is performing even though when I test these items through our LNS I get the expected outcomes (from what I can see, example a confirmed packet sends a confirmed packet etc)

Environment I am using this library so that I am compatible with ESP32 and ESP-IDF, however I am here since the compliance source files should operate the same in this environment as an Arduino environment. I am operating in EU868, AU915 and US915 regions using a custom made board (note we have no issues with any LoRaWAN components, my issues appear to be entirely with entering compliance mode and completing the automated test)

Additionally, Looking through this document: https://lora-alliance.org/wp-content/uploads/2020/11/lora_alliance_certification_deep_dive.pdf It would appear that the transmission interval needs to be 5s and it needs to exit after 192 frames. I added some code to exit compliance testing once we have had 192 uplinks without a downlink. The way I have setup the compliance mode has the transmission interval at 2s which may be too fast.

I've added in the linkcheck code listed here and it works great (at least during my validation through our LNS).

Since the ESP32 library I use goes into deepsleep, I have implemented a simple while loop to prevent the MCU from going into deepsleep during Test Mode activation using this before it goes to sleep to catch it during Test mode and go to sleep once test exits: while(LMIC_Compliance.state != 0) //While compliance mode is not in idle state { compliance_run(); //Print Variables related to compliance to keep an eye on things //printf("LMIC_Compliance.state %d", LMIC_Compliance.state); }

I also see that the join-request function exits compliance activation mode after joining, I don't know if this is correct? (I don't have the redwoodcomms module myself unfortunately, so my testing is limited...)

Summarizing into questions:

  1. How do I make sure the transmission time is 5s rather than ~2s
  2. Is there more required in the join-request function?
  3. Has this library passed 1.0.3 lorawan compliance testing using a redwoodcomms device
  4. Has anyone got any documentation on whats involved in test that is performed for 1.0.3, i can only find documentation for 1.0.4

Cheers, Dylan

DylanGWork commented 1 year ago

Update:

I am still getting some fails, some surprise me such as the confirmed uplink re-transmission as I can test that independently with the LNS and it appears to work, others I'm unsure if they are included in this library such as section 5.9 RX window test?

These were the results I got using LoRaWAN 1.0.3 on EU868: <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">

 

ITEMS | VERDICT | SUB VERDICT -- | -- | -- 1 Activation and Deactivation Pre and Post test | PASS |   1.1 Certification Application Activation |   | PASS 1.2 Certification Application Deactivation |   | PASS 2 Over the Air Activation | FAIL |   2.1 Pre-Join Behaviour |   | FAIL 2.2 Join Accept with DLSettings |   | PASS 2.3 Join Accept with Delay Settings on RX2 window |   | PASS 2.4 Join Accept with CFList |   | PASS 2.5 DevNonce Verification for Join Request |   | PASS 3 Activation by Personalization | PASS |   3.1 Activation by Personalization |   | PASS 4 Certification Application Functionality | FAIL |   4.1 Default Setting Test | PASS |   4.1.a Channel Plan and Usage |   | PASS 4.1.b Cryptograph | PASS |   4.1.b.i AES Encryption |   | PASS 4.1.b.ii Message Integrity Code |   | PASS 4.1.c Downlink Error Rate |   | PASS 4.1.d Receive Window Timing |   | PASS 4.1.e Frame Sequence Number | PASS |   4.1.e.i Uplink Sequence Number |   | PASS 4.1.e.ii Downlink Sequence Number |   | PASS 4.1.e.iii Downlink Sequence Number Rollover |   | PASS 4.2 Confirmed Frames | FAIL |   4.2.a Confirmed Uplinks |   | PASS 4.2.b Confirmed Uplink Retransmission |   | FAIL 4.2.c Confirmed Downlinks |   | PASS 4.2.d Confirmed Downlink Retransmission |   | PASS 5 MAC Command Tests | FAIL |   5.1 DevStatusReq |   | PASS 5.2 NewChannelReq |   | PASS 5.3 DlChannelReq |   | PASS 5.4 RXParamSetupReq |   | PASS 5.5 RXTimingSetupReq |   | PASS 5.6 TxParamSetupReq |   | PASS 5.7 LinkCheckReq |   | PASS 5.8 LinkADRReq | FAIL |   5.8.a ADR Bit |   | PASS 5.8.b TxPower |   | PASS 5.8.c Upink Channel Management | FAIL |   5.8.c.i Required data rates |   | PASS 5.8.c.ii Optional data rates |   | FAIL 5.8.c.iii Unsupported data rates |   | FAIL 5.8.c.iv Channel Mask and Disable all Channels |   | PASS 5.8.d Redundancy |   | PASS 5.8.e Data Rate Decay |   | PASS 5.8.f Command Block Channel Management | PASS |   5.8.f.i Successful LinkADRReq block |   | PASS 5.8.f.ii Unsuccessful LinkADRReq block |   | PASS 5.9 RX window Test | FAIL |   5.9.a RX1 Receive Window Test |   | FAIL 5.9.b RX2 Receive Window Test |   | FAIL 5.9.c RX1 and RX2 Simultaneous Frames |   | FAIL 5.9.d RX Oversized Payload |   | FAIL 5.9.e Maximum Allowed Payload | PASS |   5.9.e.i.1 Uplink PL Max Payload via Echo |   | PASS 5.9.e.i.2 Uplink PL Oversized Payload via Echo |   | PASS 5.10 MAC commands in App-Payload(FRMPayload) | PASS |   5.10.a App-Payload Only |   | PASS 5.10.b Frame Options Only |   | PASS 5.10.c App-Payload and Frame Options |   | PASS 5.11 Incorrect MAC Commands |   | PASS