Switch to main branches and support KSP2

[HebaruSan]

Motivation

The new KSP2 metadata repos have main branches instead of master:

• https://github.com/KSP-CKAN/KSP2-NetKAN
• https://github.com/KSP-CKAN/KSP2-CKAN-meta

We have been intending to make this same change to the existing repos for a while.

Also KSP2 is in early access and needs a metadata bot.

Changes

• Now all usages and mentions of a master branch are changed to main
• Now the Scheduler, Inflator (depends on Dockerfile change in KSP-CKAN/CKAN#3797), Indexer, Adder, Mirrorer, and DownloadCounter and their queues are duplicated for KSP2. (I just prefixed everything with KSP2- or ksp2_.)

We will have to be careful about how we merge this, as it will require merging KSP-CKAN/CKAN#3797 first and renaming the master branches of NetKAN and CKAN-meta. It might be best to make a public announcement of 1–3 days of downtime ahead of time.

In @techman83's testing, GitHub automatically redirects the CKAN client's repo URL to the new branch. 🎉

Open questions

• [ ] How should we handle the Webhooks? The cleanest structure would be two separate instances, since we could swap out the queues and repos, but only one of them could listen on the HTTP(S) port(s).

[techman83]

There a couple of challenges here

• How do clients handle getting metadata from main if they're pointed at master?
• Do we need to double our infrastructure to support KSP2, or should we consider some smarts to handle multi game?

We'd very likely have to scale up the instance size to handle just the base load.

[HebaruSan]

We've got all these nice micro-services with the repo and queue set up as parameters. If we can re-use them by tweaking those parameters, that seems like a huge win.

[HebaruSan]

It's also worth noting that the KSP2 repos are going to be tiny by comparison for quite a while. I don't think the new services will have anything like the same load as the current ones for, what, 1–2 years? And by that time, we may have frozen enough of the idle and abandoned KSP1 mods to compensate.

[techman83]

It's also worth noting that the KSP2 repos are going to be tiny by comparison for quite a while. I don't think the new services will have anything like the same load as the current ones for, what, 1–2 years? And by that time, we may have frozen enough of the idle and abandoned KSP1 mods to compensate.

It's true. If we shape it up nicely, it'd be reasonably doable to onboard other games as well. Looking over the code, the bulk of the work would be around the message handler (at least for the indexing component).

I see two primary areas that we'd probably need to figure out

• Queue handling - I think we can probably leave the message handling as is to a large degree, and move the queue handling out of the CLI into a class, something that can process the messages into the different games and run them through the queues.
• Configuration - Our common.queue and entry points weren't really designed for multi game.
  • Maybe meta/kan-remotes can be a mutli var, that comes in as key pairs. ie ksp=http://github.com/ksp-ckan/ckan-meta
  • Class for game repos? I can't think of a neat pattern here, but some refactoring of how we interact with repos is probably required. Something like common.get_repos('ksp') maybe?

[HebaruSan]

• How do clients handle getting metadata from main if they're pointed at master?

Noting for posterity where you answered your own question 😉 :

[techman83]

Answering my own questions is my favourite :laughing:

[HebaruSan]

Replaced by #283.