Cannot export multiple devices using the CLI

[Brammers45]

Summary

The only way listed in the documentation for exporting devices is one at a time using the get command with a few optional flags. There's no documented method for exporting a whole applications devices, and the way that should work doesn't.

Also is there a way to keep the payload formatter attached to the device through export and import, else every device will have to be manually configured after import?

Steps to Reproduce

Using the get command listed in the documentation ttn-lw-cli end-devices get migration-testing test-lux --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings works great for a singular device but cannot be used for multiple devices as the device-id field is required so get can only be used for one device at a time. Next I tried using: ttn-lw-cli end-devices search migration-testing --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings all of these flags are listed under the --help section for the search command yet this command returns error:pkg/rpcmiddleware/validator:field_mask_paths (forbidden path(s) in field mask) forbidden_paths=[frequency_plan_id lorawan_phy_version lorawan_version mac_settings root_keys supports_join] correlation_id=46c16aee396240288137d873ee5aacee

Current Result

Using search does return all devices inside an application but the optional flags don't work, which means search can't be used for the import/export of devices

Expected Result

Using ttn-lw-cli end-devices search <application-name> --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings should provide a JSON with all devices inside an application with all the necessary fields to import into another instance of TTS. There should also be an optional flag to include the payload formatters for each device, so there's no manual configuration required per device upon import.

Relevant Logs

No response

URL

No response

Deployment

The Things Stack Open Source (self-hosted)

The Things Stack Version

3.26.0

Client Name and Version

No response

Other Information

No response

Proposed Fix

No response

Contributing

• [X] I can help by doing more research.
• [ ] I can help by implementing a fix after the proposal above is approved.
• [ ] I can help by testing the fix before it's released.

Code of Conduct

• [X] I agree to follow TTN's Community Code of Conduct.

[KrishnaIyer]

This actually isn't in the scope of ttn-lw-cli but should be possible with the migration tool https://www.thethingsindustries.com/docs/the-things-stack/migrating/migrating-between-tts-distributions/#migrate-end-devices-and-gateways

Have you tried that?

[Brammers45]

I originally tried the migration tool first before using the CLI but Migration tool is for TTSv2 to TTSv3, not v3 to v3. Several flags needed show up as errors: Error: error:go.thethings.network/lorawan-stack-migrate/pkg/export:export (export device test-lux) error:go.thethings.network/lorawan-stack-migrate/pkg/export:export (export device test-lux) device_id=test-lux correlation_id=93f84959d4f143b19a109f4024596b36 — error:pkg/rpcmiddleware/validator:field_mask_paths (forbidden path(s) in field mask) forbidden_paths=[lora_alliance_profile_ids.vendor_profile_id lora_alliance_profile_ids.vendor_id serial_number last_seen_at] correlation_id=f11c2dab6fd04d349d175c898b24f9a7

[KrishnaIyer]

I originally tried the migration tool first before using the CLI but Migration tool is for TTSv2 to TTSv3, not v3 to v3.

That's not true. The migration tool supports TTS to TTS (v3) migrations. See https://github.com/TheThingsNetwork/lorawan-stack-migrate#the-things-stack

[Brammers45]

Sorry for the misconception. I was told the migration tool wouldn't work when I posted this issue on the TTN forum. I've followed that guide through step by step and ended up with the error posted above.

[KrishnaIyer]

Ok for future issues with the migration tool, please create an issue in the lorawan-stack-migrate repository. For this particular issue, there seems to be a version mismatch between your TTS and migration tool versions. Which versions are you using for TTS and the migration tool?

[Brammers45]

TTS 3.17.1 and Migration tool 0.9.0

[KrishnaIyer]

Ok so the issue is that v3.17.1 is 10 minor versions older than the latest v3.27.0 and the migration tooling is usually compatible with the previous two versions of TTS. You will have to downgrade the migration tooling and try one of the older versions I guess.

[Brammers45]

I've had a look at this but there's no documentation I've found which states what version of the migration tool works with which version.

[KrishnaIyer]

Yes unfortunately the tool and TTS do not follow the same release cycle. The only (bad) suggestion I have is that you can progressively downgrade the migration tool versions and try.

[Brammers45]

Coming at it from a different approach and still facing issues, so I'm putting together a powershell script to export all the details for all the sensors into a big JSON that I can import into 3.26.0. The version I am exporting from is 3.17.0 and the command listed in the documentation that should work is ttn-lw-cli end-devices get migration-testing lux-sens --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys Which returns the JSON: { "ids": { "device_id": "lux-sens", "application_ids": { "application_id": "migration-testing" }, "dev_eui": "00137A10000032F0", "join_eui": "C3B48C7549EF46D1", "dev_addr": "001565ED" }, "created_at": "2023-09-28T09:09:14.984Z", "updated_at": "2023-09-28T09:09:29.651865300Z", "network_server_address": "uk-nmuk-s0311.nissaneu.nmcorp.nissan.biz", "join_server_address": "uk-nmuk-s0311.nissaneu.nmcorp.nissan.biz", "lorawan_version": "MAC_V1_0_2", "lorawan_phy_version": "PHY_V1_0_2_REV_B", "frequency_plan_id": "EU_863_870", "supports_join": true, "root_keys": { "app_key": { "key": "B025343D151E9D3F66A1D90A1B4EBDC2" } } } When importing into 3.26.0 I get the following error: { "code": 3, "message": "error:pkg/util/io:json_token (invalid JSON token)", "details": [ { "@type": "type.googleapis.com/ttn.lorawan.v3.ErrorDetails", "namespace": "pkg/util/io", "name": "json_token", "message_format": "invalid JSON token", "correlation_id": "7c41a6d05251425fab917dbc9ab35ebd", "code": 3 } ] }

If i trim the JSON to fit exactly like what's listed in the documentation: { "ids": { "device_id": "lux-sens", "dev_eui": "00137A10000032F0", "join_eui": "C3B48C7549EF46D1" }, "name": "Light Sensor", "description": "On my desk", "lorawan_version": "MAC_V1_0_2", "lorawan_phy_version": "PHY_V1_0_2_REV_B", "frequency_plan_id": "EU_863_870_TTN", "supports_join": true, "root_keys": { "app_key": { "key": "B025343D151E9D3F66A1D90A1B4EBDC2" } } }

It returns the same error. Is the JSON requirements for 3.26.0 different to what's listed in the documentation? How do I get this to work?

[KrishnaIyer]

@Brammers45: What command are you running? This JSON is valid.

[dajtxx]

Regarding your question about the formatters, while experimenting today I found you can include the formatters object in the JSON fed to dev create and it will bring them in. This isn't documented so may be a bug or go away in future. But it seems easier than using up to 4 options with the dev set command.

[KrishnaIyer]

Coming back to this issue, the latest release of the migration tool (v0.12.0) should've fixed this issue. I'd recommend you to try now with the migration tooling.