TheThingsNetwork / lorawan-stack-migrate

Migrate devices from other LoRaWAN Network Servers to The Things Stack
Apache License 2.0
8 stars 3 forks source link

Error while exporting devices with migrate tool when device attributes doesn't follow `^[a-z0-9](?:[-]?[a-z0-9]){2,}$` pattern #35

Closed ymgupta closed 2 years ago

ymgupta commented 3 years ago

Summary

While migrating the devices from V2 to V3 with the migrating tool, users are getting an error invalid attributes[CustomerKey]: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$" when the device attributes don't satisfy the regex pattern.

Steps to Reproduce

  1. login to the ttnctl tool
  2. Set the attributes for a device that does not follow the regex pattern.
  3. Migrate the device with migrate tool

What do you see now?

The export of devices halted with the below error.

DEBUG [core]parsed scheme: ""                 
 DEBUG [core]scheme "" not registered, fallback to default scheme
 DEBUG [core]ccResolverWrapper: sending update to cc: {[{xxx.thethings.industries:1900  <nil> 0 <nil>}] <nil> <nil>}
 DEBUG [core]ClientConn switching balancer to "pick_first"
 DEBUG [core]Channel switches to new LB policy "pick_first"
 DEBUG [core]Subchannel Connectivity change to CONNECTING
 DEBUG [core]Subchannel picks a new address "xxx.thethings.industries:1900" to connect
 DEBUG [core]pickfirstBalancer: UpdateSubConnState: 0xc00058ff30, {CONNECTING <nil>}
 DEBUG [core]Channel Connectivity change to CONNECTING
 DEBUG [core]Subchannel Connectivity change to READY
 DEBUG [core]pickfirstBalancer: UpdateSubConnState: 0xc00058ff30, {READY <nil>}
 DEBUG [core]Channel Connectivity change to READY
 DEBUG rpc-client: call done                    auth-type=key duration=50.683032ms method=/discovery.Discovery/GetByAppID service-name=ttn-lw-migrate service-version=2.x.x
 DEBUG [core]parsed scheme: ""                 
 DEBUG [core]scheme "" not registered, fallback to default scheme
 DEBUG [core]ccResolverWrapper: sending update to cc: {[{xxx.thethings.industries:1904  <nil> 0 <nil>}] <nil> <nil>}
 DEBUG [core]ClientConn switching balancer to "pick_first"
 DEBUG [core]Channel switches to new LB policy "pick_first"
 DEBUG [core]Subchannel Connectivity change to CONNECTING
 DEBUG [core]Subchannel picks a new address "xxx.thethings.industries:1904" to connect
 DEBUG [core]pickfirstBalancer: UpdateSubConnState: 0xc000b246a0, {CONNECTING <nil>}
 DEBUG [core]Channel Connectivity change to CONNECTING
 DEBUG [core]Subchannel Connectivity change to READY
 DEBUG [core]pickfirstBalancer: UpdateSubConnState: 0xc000b246a0, {READY <nil>}
 DEBUG [core]Channel Connectivity change to READY
 DEBUG rpc-client: call done                    auth-type=key duration=35.943574ms method=/handler.ApplicationManager/GetDevice service-name=ttn-lw-migrate service-version=2.x.x
  INFO Clearing device keys                     dev_eui=7805620395862190 device_id=test_device_3416
 DEBUG [core]Channel Connectivity change to SHUTDOWN
 DEBUG [core]Subchannel Connectivity change to SHUTDOWN
 DEBUG [core]Channel Connectivity change to SHUTDOWN
 DEBUG [core]Subchannel Connectivity change to SHUTDOWN
 DEBUG [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
 DEBUG [transport]transport: loopyWriter.run returning. connection error: desc = "transport is closing"
Error: error:go.thethings.network/lorawan-stack-migrate/cmd:invalid_fields (invalid fields for device `test-device-3416`)
error:go.thethings.network/lorawan-stack-migrate/cmd:invalid_fields (invalid fields for device `test-device-3416`)
    dev_eui=7805620395862190
    device_id=test-device-3416
--- error:pkg/errors:validation (invalid `attributes[CustomerKey]`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$")
    field=attributes[CustomerKey]
    reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$"
    name=EndDeviceValidationError

What do you want to see instead?

Export of the device should be successful without any issues.

Environment

TTI V2 SaaS ttn-lw-migrate tool v0.7.0

How do you propose to implement this?

Can we handle this case at the migration tool in a similar way it is handling the device id constraints (updating _ occurrences with -) to enhance the user experience?

As most of the users are migrating from TTI V2/TTN V2, it would avoid the manual intervention of the users in deleting and re-adding these attributes for the devices in this case.

How do you propose to test this?

...

Can you do this yourself and submit a Pull Request?

No

AnyTimeTraveler commented 2 years ago

I am having the same problem.

Summary

My application has device ids that are just numbers.

Examples:

Steps to Reproduce

  1. Have an application in the TTN v2 stack with devices who's names are just two numbers
  2. Try to use this tool to export to the TTN v3 stack

What do you see now?

simon@workstation:(w)~^master ±
% ttn-lw-migrate application --source ttnv2 "wildschreck" --ttnv2.with-session=false
ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
  INFO Clearing device keys                     dev_eui=XXXXXXXXX device_id=10
Error: error:go.thethings.network/lorawan-stack-migrate/cmd:invalid_fields (invalid fields for device `10`)
error:go.thethings.network/lorawan-stack-migrate/cmd:invalid_fields (invalid fields for device `10`)
    device_id=10
    dev_eui=XXXXXXXXXX
--- error:pkg/errors:validation (invalid `ids`: embedded message failed validation)
    reason=embedded message failed validation
    name=EndDeviceValidationError
    field=ids
--- error:pkg/errors:validation (invalid `device_id`: value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$")
    name=EndDeviceIdentifiersValidationError
    field=device_id
    reason=value does not match regex pattern "^[a-z0-9](?:[-]?[a-z0-9]){2,}$"

What do you want to see instead?

A successful export.

simon@workstation:(w)~^master ±
% ttn-lw-migrate device --source ttnv2 "1355" --ttnv2.with-session=false
ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
  INFO Clearing device keys                     dev_eui=XXXXXXXX device_id=1355
{"ids":{"device_id":"1355","application_ids":{"application_id":"wildschreck"},"dev_eui":"XXXXXXXX","join_eui":"XXXXXXX"},"created_at":"0001-01-01T00:00:00Z","updated_at":"0001-01-01T00:00:00Z","name":"1355","lorawan_version":"MAC_V1_0_2","lorawan_phy_version":"PHY_V1_0_2_REV_B","frequency_plan_id":"EU_863_870_TTN","supports_join":true,"root_keys":{"app_key":{"key":"XXXXXXXXX"}},"mac_settings":{"supports_32_bit_f_cnt":true,"status_time_periodicity":"0s","status_count_periodicity":0}}

Environment

Ubuntu 20.04 Application installed via snap.

Output of command ttn-lw-migrate version:

ERROR: ld.so: object 'libgtk3-nocsd.so.0' from LD_PRELOAD cannot be preloaded (failed to map segment from shared object): ignored.
Migrate from other LoRaWAN network servers to The Things Stack: ttn-lw-migrate
Version:             0.7.0
Build date:          2021-09-27T14:43:12Z
Git commit:          ca2d9a2
Go version:          go1.16.8
OS/Arch:             linux/amd64

How do you propose to implement this?

Remove the or adjust the regex that filters valid device_ids.

If that's not possible, then request a different name be given to conform to the required naming scheme in the new TTN stack (if it's that, which requires that device names obey this regex)

How do you propose to test this?

I can just retry the export command again and see if it works.

Can you do this yourself and submit a Pull Request?

I will try to remove the regex in a fork and see what happens...

Actually this regex comes from the things stack itself and can't be changed by me.

It seems that this is out of my hands:

https://github.com/TheThingsIndustries/lorawan-stack-docs/blob/e11b0edc9397a0725af0972eb40b1eea0a5ed134/doc/content/reference/id-eui-constraints/index.md

KrishnaIyer commented 2 years ago

@AnyTimeTraveler: Your issue was fixed with a previous PR. Please try with the latest version.

@ymgupta: Is the original issue still necessary? Are users still using this tool for TTN v2 migration?

ymgupta commented 2 years ago

Hi @KrishnaIyer,

In recent times we don't see any TTN V2 users using this tool/reporting this issue to us. However, there are TTI V2 customers who will be using this tool in their migration to TTS V3.

Feel free to close it for now. If we see the need to implement this, we will reopen.