feat: Consider additionally supporting downloading Northstar from Thunderstore

[GeckoEidechse]

What feature would you like added? Viper currently downloads Northstar from GitHub but we also upload Northstar to Thunderstore as well. In fact Thunderstore version gets auto-upload by CI on release.

Why should this feature be added? The reason for supporting Thunderstore Northstar download is a bit of selfish one. We're planning to also auto-upload release candidates to their own package on Thunderstore (NorthstarReleaseCandidate vs Northstar). By supporting Thunderstore download, it would be really easy to then add support for selecting a release channel in Viper. So player could then pick whether they want to be in "stable" or "release candidate" channel. "stable" would obviously be the default while "release candidate" would have to be changed to manually. That way when we enable auto-upload release candidates we can then point playtesters to just select "release candidate" channel in Viper to playtest the newest release candidates which is way easier for them to do compared to having to manually download a release-candidate from GitHub actions everytime we do a new one. Especially as Viper already supports auto-download on update.

Additional Info I'd still keep support for GitHub and then have an option to select download source. Whether GitHub or Thunderstore should be the default is a decision I'll leave up to you. Currently it makes no difference :P

[0neGal]

I completely agree with everything and I'll have a look at implementing this, it'll just be a switch for selecting where the Northstar release will be fetched and downloaded from, which would then have GitHub, Thunderstore Stable and Unstable.

I think "release candidate" can be confusing than something like "unstable", while they aren't exactly the same thing, explaining to a user how releases are drafted and tested is a lot just to explain that it's a less stable but more upstream version.

Feel free to lmk if you think something else is better...

As for implementing this, just to confirm, the Zip archive on Thunderstore is identical in layout to the GitHub release, correct? If so we only actually need to update some stuff in requests.js and it would be able to download from there (not counting implementing a switch for picking the option, just purely getting it to install...)

Whether GitHub or Thunderstore should be the default is a decision I'll leave up to you. Currently it makes no difference :P

Personally I think Thunderstore should be default...

By supporting Thunderstore download, it would be really easy to then add support for selecting a release channel in Viper.

I do wonder, how does our code for checking if Northstar is up to date, work with rc suffixed releases? I assume it just assumes it's always outdated, so might have to revisit that...

[Alystrasz]

Why bothering creating GitHub, Thunderstore Stable or Unstable channels, when we can just fetch all available versions from Thunderstore and let users choose themselves (à-la-Minecraft Launcher, if it still works this way)?

@GeckoEidechse release notes are not published on Thunderstore, only on GitHub releases, right?

[0neGal]

Why bothering creating GitHub, Thunderstore Stable or Unstable channels, when we can just fetch all available versions from Thunderstore and let users choose themselves (à-la-Minecraft Launcher, if it still works this way)?

It really wouldn't take a lot to implement, and if Thunderstore is down it's always good to have an alternative, and likewise with GitHub.

[GeckoEidechse]

which would then have GitHub, Thunderstore Stable and Unstable.

Sounds like a good idea. Probably best to hint somewhere in UI that there's not a difference between "Thunderstore stable" and "GitHub", maybe display the most current version next to download source when selecting download source?

I think "release candidate" can be confusing than something like "unstable", while they aren't exactly the same thing, explaining to a user how releases are drafted and tested is a lot just to explain that it's a less stable but more upstream version. Feel free to lmk if you think something else is better...

Don't really have an idea so I'll leave it up to you :P

We're planning to also do a rolling release "git main" package in the future but we still need to figure out how to do infrastructure for that. I guess that one could be called "dev" then ^^

As for implementing this, just to confirm, the Zip archive on Thunderstore is identical in layout to the GitHub release, correct?

Sadly no but almost. When downloading from Thunderstore the resulting zip contains a folder called Northstar in its root. Going into that folder the files are the same as those from GitHub. (On Thunderstore we do not ship ns_startup_args{_dedi}.txt as it would cause r2mm to overwrite each Northstar update. We still have to drop them from GitHub release. NorthstarLauncher will self generate those files if they do not exist.)

By supporting Thunderstore download, it would be really easy to then add support for selecting a release channel in Viper.

I do wonder, how does our code for checking if Northstar is up to date, work with rc suffixed releases? I assume it just assumes it's always outdated, so might have to revisit that...

I'd go for picking newest in that channel. So if newest unstable is older then newest stable, still stick with newest unstable. For version checking on NorthstarMasterServer we used the semver package, see https://github.com/R2Northstar/NorthstarMasterServer/pull/55/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R25

 

@GeckoEidechse release notes are not published on Thunderstore, only on GitHub releases, right?

Yup, so release note fetching would stay the same regardless of selected channel. We currently don't really do release notes for release candidates, though release note drafts should be publicly visible.

[0neGal]

Don't really have an idea so I'll leave it up to you :P

We're planning to also do a rolling release "git main" package in the future but we still need to figure out how to do infrastructure for that. I guess that one could be called "dev" then ^^

How about "GitHub (Stable)", "Thunderstore (Stable)", "Thunderstore (Canary)" (rc builds), and "Thunderstore (Unstable)"

I'd go for picking newest in that channel. So if newest unstable is older then newest stable, still stick with newest unstable. For version checking on NorthstarMasterServer we used the semver package, see https://github.com/R2Northstar/NorthstarMasterServer/pull/55/files#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519R25

Hmm, we'll just modify what requests.getLatestNsVersion() spits out (depending on the release source), and compare versions slightly differently then...

[GeckoEidechse]

How about "GitHub (Stable)", "Thunderstore (Stable)", "Thunderstore (Canary)" (rc builds), and "Thunderstore (Unstable)"

I'd use development over unstable, as unstable just feels too vague to me. Otherwise seems fine to me ^^

[0neGal]

I'd use development over unstable, as unstable just feels too vague to me. Otherwise seems fine to me ^^

The idea is rather to really empathize that it's not really stable... Developer may not exactly imply the stability of it to some people... Idk. We could also rename Canary to Testing, anything is really possible of course.

I'm not sure what better words there really are to describe it really. We could also go all the way with the Discord route and have Stable, PTB (Public Test Build), and Canary. And then note that Canary is not stable, PTB is meant for testing, and Stable is the recommended build.

[GeckoEidechse]

Maybe combine the two words and use unstable, development? idk ^^

[0neGal]

Maybe combine the two words and use unstable, development? idk ^^

Considering everything else is a single short word and nothing more, that'd be a very long line...

GitHub (Stable) Thunderstore (Stable) Thunderstore (PTB) Thunderstore (Unstable Development)

Keep in mind the UI this would be in, though, now that I think of it, In-Development might work, aka:

GitHub (Stable) Thunderstore (Stable) Thunderstore (PTB) Thunderstore (In-Development)

Ofc whether it's PTB or Canary, or whatever doesn't matter much and if nobody has an opinion I'll figure out what fits best, from my opinion that is...

[GeckoEidechse]

So turns out Thunderstore only allows MAJOR.MINOR.PATCH versioning (c.f.: https://github.com/thunderstore-io/Thunderstore/issues/634). This means stuff like v1.9.1-rc1 wouldn't work.

To circumvent this, we are considering doing MAJOR.MINOR.100*PATCH+RCNUMBER, i.e. v1.9.1-rc1 -> v1.9.101 for the time until Thunderstore changes their requirements (if ever).

At least for Viper this means a change in versioning detection library won't be needed for now :P

[GeckoEidechse]

The release candidate Thunderstore package (manually uploaded for now): https://northstar.thunderstore.io/package/northstar/NorthstarReleaseCandidate/

[GeckoEidechse]

At least for Viper this means a change in versioning detection library won't be needed for now :P

One pain point I just realised is that the Northstar mods version will still say v1.9.1-rc1 (the way they should). Meaning checking if version installed is newest on Viper will be a bit of pain. Basically you'd have to implement the same calculation as above to go from "proper rc semver release number" to "thunderstore release number" :c

[0neGal]

So turns out Thunderstore only allows MAJOR.MINOR.PATCH versioning (c.f.: thunderstore-io/Thunderstore#634). This means stuff like v1.9.1-rc1 wouldn't work.

I suggest waiting until that is fully sorted, as it would not only be a waste of work getting it solved and then having to scrap it later, but it'll also likely require a bit of work for it to function properly.

One pain point I just realised is that the Northstar mods version will still say v1.9.1-rc1 (the way they should). Meaning checking if version installed is newest on Viper will be a bit of pain. Basically you'd have to implement the same calculation as above to go from "proper rc semver release number" to "thunderstore release number" :c

Again, it seems easier to wait till everything is solved with the whole suffix...

[GeckoEidechse]

So based on discussion in discussion in https://github.com/thunderstore-io/Thunderstore/issues/634 as well as in direct messages with Mythic I'm under the impression that this will take quite some time to implement. In particular it's not yet clear whether there will ever be suffix support as it might just be changed to add support different release channels, which of course takes time.

From the Northstar side of things, we're planning to add auto-uploading rc builds to separate package some time this weekend. Given the limitations by Thunderstore we'll stick to encoded version numbers e.g. v1.9.101 as mentioned above.

For Viper I guess the best course of action is to just add support for downloading the stable package from Thunderstore for now, preferably in a more generic way to allow for easy extending to other download sources in the future. After that I could look into downloading release candidate package in Viper. Maybe to keep things simpler from Viper side of things we just take over the version number of Thunderstore i.e. also display v1.9.101 and force Northstar reinstall on switching download source to avoid having an rc like v1.9.101 block a technically newer version like v1.9.2 (stable).

[0neGal]

So the mod.json's in the Northstar mods will also have the v1.9.101 version numbers as well? And no rc suffix anywhere? If so this would be pretty easy to implement :)

[GeckoEidechse]

So the mod.json's in the Northstar mods will also have the v1.9.101 version numbers as well? And no rc suffix anywhere? If so this would be pretty easy to implement :)

Oh right, there's the catch. Yeah no it does not have that due to the way our CI is build up, i.e. we first package Northstar and then do Thunderstore specific stuff.

We could go in and re-modify it in Thunderstore step but I'm not really a fan of that due to it misrepresenting the version of the package even more.

I guess the best solution would be to have a "translation function" in Viper that takes the the package type (GitHub, Thunderstore stable, Thunderstore unstable) and returns the translated version number. So if installed version is Thunderstore unstable and mod.json says v1.9.1-rc1, then return value would be v1.9.101 which would then get checked for update on Thunderstore (or vice versa, not exactly sure how Viper does update check)

Btw, if you need a package to check against for unstable/release-candidate, I uploaded one manually for now https://northstar.thunderstore.io/package/northstar/NorthstarReleaseCandidate/ ^^

[0neGal]

I guess the best solution would be to have a "translation function" in Viper that takes the the package type (GitHub, Thunderstore stable, Thunderstore unstable) and returns the translated version number. So if installed version is Thunderstore unstable and mod.json says v1.9.1-rc1, then return value would be v1.9.101 which would then get checked for update on Thunderstore (or vice versa, not exactly sure how Viper does update check)

Hmm, we could also just check if -rc is in the name, and replace it with a 0 so v1.9.1-rc1 becomes v1.9.101, assuming that would always return correctly?

[GeckoEidechse]

Hmm, we could also just check if -rc is in the name, and replace it with a 0 so v1.9.1-rc1 becomes v1.9.101, assuming that would always return correctl

Assuming we don't have more than 9 release-candidates that should work just fine ^^

Otherwise we might have to go for zero-padding rc number to 2 digits and then just stripping the -rc.

[GeckoEidechse]

Support for auto-publishing release-candidates to Thunderstore is nearly done btw ;)

https://github.com/R2Northstar/Northstar/pull/313

[0neGal]

Just to clarify, both the Thunderstore version, and the mod.json files will all have v1.9.101 in them when it's an RC build? And if there's more than one build it becomes v1.9.102? When it reaches 10 builds does it go to v.19.110?

If so, this shouldn't be hard to implement!

[GeckoEidechse]

Just to clarify, both the Thunderstore version, and the mod.json files will all have v1.9.101 in them when it's an RC build?

No, as stated before that's kinda difficult to implement with our CI and would mess with other things, see https://github.com/0neGal/viper/issues/142#issuecomment-1183584130 :P

And if there's more than one build it becomes v1.9.102? When it reaches 10 builds does it go to v.19.110?

This part, very much yes. At least that's the plan, cause currently it doesn't do it yet but I will fix it before we reach 10 release candidates of the same version (which has never happened so far anyway :P)

[0neGal]

No, as stated before that's kinda difficult to implement with our CI and would mess with other things, see #142 (comment) :P

Right, brain bad, and I forgot lol, I would say, just replace -rc with 0, but... If we get over 9 releases, how would we reliably detect whether to replace it with 0, 1 and so on...

[GeckoEidechse]

If we get over 9 releases, how would we reliably detect whether to replace it with 0, 1 and so on...

You mean passing 9 on the last digit of semver, so like e.g. v1.9.10-rc2? That would be v1.9.1002. So like last two digit are release candidates, the rest is actual patch release number ^^

Well the correspondence is basically 0.0.1-rc10 <-> 0.0.110, so basically

[0neGal]

If we get over 9 releases, how would we reliably detect whether to replace it with 0, 1 and so on...

You mean passing 9 on the last digit of semver, so like e.g. v1.9.10-rc2? That would be v1.9.1002. So like last two digit are release candidates, the rest is actual patch release number ^^

Well the correspondence is basically 0.0.1-rc10 <-> 0.0.110, so basically

ah, so we actually just remove the -rc part, and pad it if it's below 9, sure hope you'll never reach 100 rc releases lol

[GeckoEidechse]

https://github.com/R2Northstar/Northstar/pull/313 has now been successfully merged and the Northstar built on Thunderstore from a test run from building v1.9.1-rc4 can be found here: https://northstar.thunderstore.io/package/northstar/NorthstarReleaseCandidate/1.9.104/

[GeckoEidechse]

Thinking about this again, it might even be better to drop support for downloading from GitHub entirely and going for just downloading from Thunderstore.

[0neGal]

Any reason for dropping support entirely over just changing which one it uses by default?

[GeckoEidechse]

Any reason for dropping support entirely over just changing which one it uses by default?

Mostly to avoid having to maintain duplicate logic I guess. Another reason is that the GitHub release still ships ns_startup_args.txt and ns_startup_args_dedi.txt which Thunderstore does not.

I've been planning to drop it from GitHub release for a while now but not sure how mod managers will react to those files missing from release zip. (The game generates them on launch if they don't exist)

Also there's not really any other difference between the two so there's no point in having both (unless the idea is to have a fallback in case one of the two services is down).

[0neGal]

... but not sure how mod managers will react to those files missing from release zip.

For Viper, we actually just write the file when changes (settings) are made or similar, whether it exists already or not makes no difference, so at least for Viper it'd have no effect.

unless the idea is to have a fallback in case one of the two services is down

Chances of GitHub going down isn't that big, chances of Thunderstore is bigger and a possibility, unless it's annoying to maintain, or there's incompatibilities between the two, I don't see much reason to switch over to Thunderstore and not keep GitHub as an alternative.

If you would've preferred GitHub no longer had releases then yeah, we'd switch over to Thunderstore only, but otherwise, I don't see the reason.

[GeckoEidechse]

Fair enough ^^

[Alystrasz]

Why bothering creating GitHub, Thunderstore Stable or Unstable channels, when we can just fetch all available versions from Thunderstore and let users choose themselves (à-la-Minecraft Launcher, if it still works this way)?

It really wouldn't take a lot to implement, and if Thunderstore is down it's always good to have an alternative, and likewise with GitHub.

Northstar release v1.9.5 was released yesterday, and was provoking screen freezes and other issues. Some Viper users (having auto-updating enabled) couldn't easily retrograde to v1.9.4; a version selector (picking binaries from both GitHub and Thunderstore if you will) could solve this issue, and easily allow people to play with older Northstar builds.

[GeckoEidechse]

Regardless to what should be default, Viper currently still does not support installing Northstar from Thunderstore, right?

Or at least I couldn't find anything related in the UI unless I'm mistaken. ^^"

[0neGal]

Viper currently still does not support installing Northstar from Thunderstore, right?

It does not no, been busy with life and all, and haven't had much time to dedicate to adding minor things like this. It is however something I intend to add, sooner or later, among a couple other features.

[GeckoEidechse]

It does not no, been busy with life and all, and haven't had much time to dedicate to adding minor things like this.

That is totally ok. Setting priorities is important and there are things out there that are far more important than Northstar <3 Just wanted to make sure I didn't miss anything ^^"