search

TheThingsIndustries / lorawan-stack-docs

Documentation for The Things Stack
Apache License 2.0
32 stars 65 forks source link

Migration from TTS CE to TTS C #542

Closed rish-c closed 2 years ago

rish-c commented 3 years ago

Summary

We need documentation for the scenario where a user moves from TTS Community Edition to TTS Cloud.

Why do we need this ?

Now that the community edition has been available for a few months, there are several users who start with TTN and at some point want to move to TTS Cloud.

What is already there? What do you see now?

This particular topic is not addressed on our docs.

What is missing? What do you want to see?

A section similar to other cases where we walk a user through this flow. Perhaps ttn-lw-migrate can do this.

How do you propose to document this?

Not sure, hence this issue.

Can you do this yourself and submit a Pull Request?

@KrishnaIyer, cc @NicolasMrad

htdvisser commented 3 years ago

See also https://github.com/TheThingsIndustries/lorawan-stack-support/issues/187#issuecomment-915010056

ymgupta commented 2 years ago

In recent times, we are getting quite some requests from users regarding the migration of devices between TTS CE and TTSC. Currently, there is no defined tool or documented process for this. We can provide the below steps as a workaround process until we streamline this.

It is recommended to migrate the devices without active sessions when between networks. Please refer below on how this is done. Prerequisites

  1. An application in your TTSC (The Things Stack Cloud) tenant.
  2. Setup your integrations and Payload formatters for the new application.
  3. Configure the CLI to TTS CE (Things Stack Community Edition). Ref: Command-line Interface

Migration without Active sessions: Method 1: Delete the device from TTS CE and recreate it in the new application in TTSC V3. You can script to add new devices using APIs. Ref: Adding Devices

Method 2: This method involved exporting the device information from TTS CE, making necessary modifications to the file and import to TTSC. Note:

  1. We highly recommend to always test the process on a single end device, to make sure the migration process goes as expected. Once you ensure everything goes well, you can start the migration of production overloads in small batches.
  2. The process and commands given are for migrating one single device. If you want to migrate all the devices in an application at a time, you need to script the commands.
  3. Use the latest version of the CLI (v3.18.2). Ref: Releases · TheThingsNetwork/lorawan-stack

Step 1: Export device from the TTS CE Run the below commands to export a device without the session from the TTS CE.

ttn-lw-cli end-devices get --application-id <app-id> --device-id <device-id> --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings > <device-id>_device_information.json

Note: Please make sure you update the CLI configuration file to point to the respective cluster (eu1/nam1/au1) of your The Things Stack CE where the devices are registered. You can generate the contents of the CLI configuration file for the TTS CE based on the cluster in use (eu1/au1/nam1) from the reference below and update them in the .ttn-lw-cli.yml file. Ref: Addresses

Step 2: Remove the entries of Server addresses in the JSON file

Step 3: Delete the device in the TTS CE Before you proceed with Step 4, the device needs to be deleted from the TTS CE. You can delete the device directly from the TTS CE Console or via CLI. CLI Command:

ttn-lw-cli end-devices delete --application-id <application-id> --device-id <device-id>

Step 4: Import the device in the TTS Cloud via Console Add the device to the respective Application in the TTSC via Console by importing the previously exported JSON file. Ref: Import End Devices in The Things Stack Note:

  1. Please make sure the JSON file used for import should be less than 4MB in size.
  2. Please make sure you import the devices in the appropriate cluster (eu1/eu2/au1/nam1).
  3. If you are using CLI to import the devices, please make sure you update the CLI configuration file to point to the respective cluster (eu1/eu2/nam1/au1) of your TTSC V3 tenant where the devices need to be imported.

Step 5: Gateway migration Please note that TTS CE and TTSC are connected via Packet Broker already. So, if you choose to migrate devices without active sessions, then the existing gateways will be able to handle the joins, route the data from the migrated devices while still pointing to TTS CE, and it means you need not perform any extra configuration on these gateways immediately. You can consider migrating these gateways gradually at a later stage. Note: The device needs to initiate a join after being added to the new application in TTSC V3.

Migration with Active sessions: Currently, migrating the devices with an active session from The Things Stack Community Edition (TTS CE) to The Things Stack Cloud V3 (TTSC) cannot be done automatically since they are different networks and there are certain limitations. However, with a few changes to Step 1 and Step 5 in Method 2 suggested above, you can migrate a device with session. Refer to the changes below.

Step 1: Export device from the TTS CE Run the below commands to export a device with the session from the TTS CE.

ttn-lw-cli end-devices get --application-id <app-id> --device-id <device-id> --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings --mac-state --session > <device-id>_session_information.json

Step 5: Gateway migration If the devices are to be migrated with persisting active session, then it is essential that gateways are also migrated simultaneously along with devices (site-by-site). This is because the Packet Broker routes traffic based on the DevAddr blocks assigned for the tenants and the DevAddr block of TTS CE is different from that of TTSC.

Rest of the steps stay the same as mentioned above. Please note that this is a workaround and not a standard process. We highly recommend to always test the process on a single end device, to make sure the migration process goes as expected. Once you ensure everything goes well, you can start the migration of production overloads in small batches.

It is better to document the above steps for now. Many users, however, find this a tedious process because many steps need to be done manually and are error-prone.

@htdvisser /@adriansmares, please validate the process before adding this to the documentation.

CC: @NicolasMrad

johanstokking commented 2 years ago

OK, I think what we miss is support for TTS to TTS migration in ttn-lw-migrate. It's pretty similar to ttn-lw-cli already but we can automatically delete devices after export etc, add flags etc.

So what I think we should do is:

  1. Remove V2 documentation from https://www.thethingsindustries.com/docs/getting-started/migrating/, and bury it under a legacy TTN V2 section
  2. Prioritize Migration Tool page so it's higher in the menu list. Also don't link to GitHub when clicking ttn-lw-migrate like here. People get lost on GitHub
  3. Implement TTS source in ttn-lw-migrate and document it
benolayinka commented 2 years ago

@KrishnaIyer are you maintaining ttn-lw-migrate?

KrishnaIyer commented 2 years ago

TTS repositories don't have individual maintainers. Please create an issue and a member of the product team who is available at the time will pick it up.

benolayinka commented 2 years ago

I created an issue to update ttn-lw-migrate. Once that's done, @nejraselimovic can you update the migration guide to include migration from TTS CE? Let me know if you need any help.

nejraselimovic commented 2 years ago

@johanstokking @ymgupta is it necessary for migration with an active session to remove the dev_addr field from the device.json file? I just tried to migrate a session without removing the DevAddr and it worked, and it also worked when I remove the DevAddr. When I removed the DevAddr from the device.json and imported it in TTS CH, I could still see the same DevAddr in device settings, even though I removed it from the description file. Now I'm wondering what's the catch there

johanstokking commented 2 years ago

You can technically move the device with session, but that only works if there's a gateway directly connected to the target network (in thise case TTSC). Because Packet Broker won't route it, as the DevAddr block is assigned to the source network (in this case TTSCE) only.

ymgupta commented 2 years ago

@nejraselimovic, in earlier versions of TTS, importing the JSON without removing dev_addr used to fail with an error. So, we included that step. However, with the latest version of TTS, we are also not seeing that issue. So, that step can be ignored.

nejraselimovic commented 2 years ago

@ymgupta can you please write down instructions and recommendations for ABP devices as well? I'm trying to test migration of ABP device, but I don't really understand its behavior too well

ymgupta commented 2 years ago

@nejraselimovic, below are the slight modifications needed in the above process (defined for OTAA devices) to migrate the ABP device from TTS CE to TTS Cloud.

Without session Migration: CLI Command:

ttn-lw-cli end-devices get --application-id <app-id> --device-id <device-id> --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --root-keys --mac-settings --session.dev-addr --session.keys > <device-id>_device_information.json

Instructions:

With Session Migration: CLI Command:

ttn-lw-cli end-devices get --application-id <app-id> --device-id <device-id> --name --description --lorawan-version --lorawan-phy-version --frequency-plan-id --supports-join --mac-settings --mac-state --session > <device-id>_session_information.json

Instructions:

Caution: If the existing ABP device in TTS CE does not have factory preset frequencies configured, the user might experience an uplink/downlink discrepancy (Eg: packets drop at the gateway) following the migration.

Let us know if anything is not clear or needs more information.