Support migration from the Firefly LNS

KrishnaIyer commented 11 months ago

Summary

Support migration from the Firefly LNS. Replaces #49.

Changes

Improve on the ground work in #49
Make the migration/exporting work.
Improve code reuse.

Testing

With a The Things Uno.

Curl the device using the API (to check the last frame counter)

curl "https://xxxxxxx/api/v1/devices/eui/0004A30B001BB199?auth=$FIREFLY_API_KEY"
{"device":{"updated_at":"2023-10-31T18:30:53.110455","tags":[],"skip_fcnt_check":false,"rx2_data_rate":0,"region":"EU868","override_location":false,"otaa":true,"organization_id":226,"organization":{"name":"alec","id":226,"description":null},"network_session_key":"1BA000AB1BAC28B83F2A091F7EA9ABED","name":"Migration Test OTAA","multicast_groups":[],"location":null,"inserted_at":"2023-10-31T18:11:08.544913","id":62268,"frame_counter":0,"eui":"0004A30B001BB199","device_class_id":null,"description":null,"deduplicate":false,"class_c":false,"application_session_key":"927A87F8B710479D594440227BBBDD64","application_key":"5105D918B6EEEF8A22AA92B39D9EC718","adr_limit":null,"address":"29559735"}}

This can also be checked via the UI

Export


./main firefly device 0004A30B001BB199 --invalidate-keys

{"ids":{"device_id":"eui-0004a30b001bb199","application_ids":{"application_id":"my-test-app"},"dev_eui":"0004A30B001BB199","join_eui":"1111111111111111"},"name":"Migration Test OTAA","lorawan_version":"MAC_V1_0_2","lorawan_phy_version":"PHY_V1_0_2_REV_B","frequency_plan_id":"EU_863_870","supports_join":true,"root_keys":{"app_key":{"key":"5105D918B6EEEF8A22AA92B39D9EC718"}},"mac_settings":{"rx2_data_rate_index":0,"status_time_periodicity":"0s","status_count_periodicity":0,"desired_adr_ack_limit_exponent":"ADR_ACK_LIMIT_1"},"session":{"dev_addr":"29559735","keys":{"session_key_id":"bBRMrEnueD7zAxHwX4a2RA==","f_nwk_s_int_key":{"key":"1BA000AB1BAC28B83F2A091F7EA9ABED"},"app_s_key":{"key":"927A87F8B710479D594440227BBBDD64"}},"last_f_cnt_up":10,"started_at":"2023-10-31T18:41:35.080851Z"}}

3. Import and check uplink (active session)
<img width="1510" alt="otaa-frame12" src="https://github.com/TheThingsNetwork/lorawan-stack-migrate/assets/13186113/45532a50-06f9-4020-a277-4ee222c2b7fc">

4. Rejoin on TTS
<img width="1512" alt="otaa-rejoin-tts" src="https://github.com/TheThingsNetwork/lorawan-stack-migrate/assets/13186113/a722378c-b80f-438a-9316-9ed717384973">

##### Regressions

<!-- Please indicate features that this change could affect and how that was tested. -->

NA. This is new functionality.

#### Notes for Reviewers
<!--
NOTE: This section is optional.

Motivate briefly why it is implemented this way, if that deviates from the
implementation proposal in the referenced issues.
- How should your reviewers approach this pull request?
- @mention reviewers with special requests or questions for them
-->

1. I've only tested one device for OTAA and ABP. Before merging, I will test a batch.
2. I'll add docs after we agree on the code.
3. I had one failed frame during the OTAA migration. Is this a frequency plan configuration issue? should I have used `EU_863_870_TTN` instead? I'm using a The Things Uno for testing.

DEBUG Failed to determine channel index of uplink {"ack": false, "adr": false, "adr_ack_req": false, "band_id": "EU_863_870", "bandwidth": 125000, "class_b": false, "data_rate": "lora:{bandwidth:125000 spreading_factor:7 coding_rate:\"4/5\"}", "dev_addr": "29559735", "error": "error:pkg/networkserver:uplink_channel_not_found (uplink channel not found)", "f_cnt_reset": false, "f_opts_len": 0, "f_port": 1, "frequency": 867700000, "full_f_cnt_up": 11, "grpc.method": "HandleUplink", "grpc.service": "ttn.lorawan.v3.GsNs", "last_f_cnt_up": 10, "m_type": "UNCONFIRMED_UP", "mac_count": 0, "mac_version": "MAC_V1_0_2", "major": "LORAWAN_R1", "namespace": "networkserver", "pending_session": false, "phy_payload_len": 14, "received_at": 1698777742.7678819, "request_id": "01HE3GPFDFN4CBZX9Q697JE83A", "spreading_factor": 7, "uplink_f_cnt": 11}


Does anything stand out to you here @adriansmares?

#### Checklist
<!-- Make sure that this pull request is complete. -->

- [x] Scope: The referenced issue is addressed, there are no unrelated changes.
- [ ] Documentation: Relevant documentation is added or updated.
- [x] Changelog: Significant features, behavior changes, deprecations and fixes are added to `CHANGELOG.md`.

KrishnaIyer commented 11 months ago

Retested the changes

OTAA The device sends frame 3 on Firefly, is migrated to TTS and TTS receives frame 4 onwards correctly. Rejoin works also
ABP The device sends frame 3 on Firefly, is migrated to TTS and TTS receives frame 4 onwards correctly

KrishnaIyer commented 10 months ago

This JoinEUI thing is probably quite problematic, no? This only works if there's one JoinEUI per batch? And you either migrate one device, or you migrate all devices accessible with the API key?

Yes indeed. The way I have documented this is to migrate devices in batches by creating a file with the Device EUIs and piping that into the devices command. This way, all devices in the batch will have a common JoinEUI on TTS.

TheThingsNetwork / lorawan-stack-migrate