search

TheThingsNetwork / lorawan-stack

The Things Stack, an Open Source LoRaWAN Network Server
https://www.thethingsindustries.com/stack/
Apache License 2.0
983 stars 309 forks source link

Gateway migration v2 to v3 in bulk #3155

Closed laurensslats closed 3 years ago

laurensslats commented 4 years ago

Summary

In addition to migrating devices from V2 to V3, documentation needs to be provided on migrating gateways in bulk.

Why do we need this?

Some TTN users and TTI customers have >1000 gateways, manually entering the EUIs and gateway settings is not an option.

What is already there? What do you see now?

Documentation on connecting gateways 1 by 1 Documentation on migrating devices in bulk via CLI Tool to bulk import devices via the Console

What is missing? What do you want to see?

  1. Bulk export feature in ttnctl
  2. Bulk import feature in TTS using the CLI
  3. Bulk import feature in TTS using the Console

Environment

The Things Network Stack V2 (TTN & TTI privately)

How do you propose to implement this?

  1. Documentation on manually migrating gateways (WiP: https://github.com/TheThingsNetwork/lorawan-stack/tree/doc/gateway-migration)
  2. Involve Aggelos for bulk-export gateways using ttnctl
  3. Involve Aggelos for bulk-import gateways using the cli
  4. Discuss how to enable bulk-import gateways using the console

How do you propose to test this?

Test importing gateways in bulk locally

Can you do this yourself and submit a Pull Request?

Need input from @neoaggelos, @htdvisser (and maybe @johanstokking)

johanstokking commented 4 years ago

We need to split this over several things;

  1. Bulk create gateways, potentially from an import file (like we have for end devices). This requires extending DTC with gateway templates (which it is somewhat prepared for already). Since gateways support collaborators, we also have to configure ownership as part of the bulk import
  2. Create an import file from V2 (like we have for end devices)
  3. Authenticate gateways: for gRPC, MQTT and Basic Station gateways, we need to recognize and somehow accept their legacy V2 gateway key
  4. As far as this is possible: update the gateway's address, i.e. the Basic Station LNS address
  5. Document gateway migration
htdvisser commented 4 years ago

Since gateways support collaborators, we also have to configure ownership as part of the bulk import

I'm not sure this is such a great idea, since the ownership model of v2 was quite different from what we now have in v3. I'd strongly suggest to just create these gateways under an organization (identified with a CLI flag) instead of trying to assign a dozen collaborators based on usernames that may be different in v3.

Authenticate gateways: for gRPC, MQTT and Basic Station gateways, we need to recognize and somehow accept their legacy V2 gateway key

I also don't think we should do this. Many gateways will have to be reconfigured to new server addresses anyway, so if we generate an API key on create, we can add that to the response so that the user has everything they need for reconfiguring the gateway.

johanstokking commented 4 years ago

I'm not sure this is such a great idea, since the ownership model of v2 was quite different from what we now have in v3. I'd strongly suggest to just create these gateways under an organization (identified with a CLI flag) instead of trying to assign a dozen collaborators based on usernames that may be different in v3.

Right, this is indeed not what we need to do. I meant that as part of the import, there has to be an owner dropdown, just like for create. So there's one collaborator: an org or the user.

I also don't think we should do this. Many gateways will have to be reconfigured to new server addresses anyway, so if we generate an API key on create, we can add that to the response so that the user has everything they need for reconfiguring the gateway.

The problem here is that if you import 1000 gateways, it's quite hard to return 1000 API keys. Telling the customer to go to their 1000 gateways is quite a bad news conversation.

Should we put the V2 API key in a gateway secret and allow authentication with that?

@KrishnaIyer what do you think about this one?

KrishnaIyer commented 4 years ago

The problem here is that if you import 1000 gateways, it's quite hard to return 1000 API keys

Agreed but this has to be done for Basic Station. In v2, we don't use auth for those gateways.

Authenticate gateways: for gRPC, MQTT and Basic Station gateways, we need to recognize and somehow accept their legacy V2 gateway key

So practically, this is only for MQTT since we don't support v2 gRPC. Indeed we need to add support to those ttn-account-v2.<keys> .

Should we put the V2 API key in a gateway secret and allow authentication with that?

Auth is one thing, with the special case of GPS gateways, we need to append the "Update Gateway Info" rights to update the location.

How about we set a sunset period for v2 Auth? We

This would involve only changes to the gateway model and the frontend (MQTT only?).

johanstokking commented 4 years ago
  • for the Auth, we just fetch the v3 API Key internally if the auth is ttn-account-v2.xxx and is valid for that gateway.
  • This v3 API Key is what's passed to the IS for rights check.

We can't do this. We need to use GS cluster auth for this, or don't support updating the gateway's location. But I believe that this is already supported with cluster auth.

  • give customers 6 months to rotate keys before we remove this

We all know how this goes. I'm afraid we need a permanent solution for V3 and sunset this with V4.

johanstokking commented 4 years ago

Extracted authentication to https://github.com/TheThingsNetwork/lorawan-stack/issues/3177. Let's focus only on the migration itself here.

neoaggelos commented 3 years ago

Is this still relevant, what do we need to do for this one? @KrishnaIyer

KrishnaIyer commented 3 years ago

I'm not entirely sure since the scope of this issue and the linked issues have changed quite a bit since this was created.

johanstokking commented 3 years ago

I don't think we should do this.