Automate releases to SDKMAN

[marc0der]

Hi all, on making some updates to the website to download/install Scala via SDKMAN, I told @julienrf that we at SDKMAN have a vendor API that allows publishing releases to SDKMAN. You can even use our API to broadcast your release to all SDKMAN users via CLI and Twitter.

I'd be more than happy to help set this up and get the Scala release build integrated with our APIs.

[julienrf]

Hi Marco, I think to move forward we need to interest some people responsible for making Scala releases (for Scala 2 it can be @SethTisue, @retronym, @lrytz, or @dwijnand I guess, and for Scala 3 it can be @anatoliykmetyuk, @prolativ, or maybe someone else?) in updating the release process. (So, if one of the people I mentioned are interested, please voice yourself)

Your help could be useful here to point us towards the relevant parts of the API, and possibly to draft the changes to make to the release process.

As far as I know, the release process is documented here for Scala 3 (important file is this one). I am not aware of something equivalent for Scala 2.

I hope that helps!

[SethTisue]

The Scala 2 release process is documented at https://github.com/scala/scala-dev/blob/scala-dev/.github/ISSUE_TEMPLATE/release.md

[marc0der]

Thanks for getting back to me @julienrf and @SethTisue.

While reading the Scala 2 release process, it wasn't clear when you generate the zip distribution. And, is this performed by the SBT native packager plugin? I've meant to write an SBT plugin for SDKMAN releases for some time. Would this help you make things easier @SethTisue?

Any guidance from the Scala 3 side would be greatly appreciated, too @anatoliykmetyuk @prolativ?

Thanks so far!

[SethTisue]

While reading the Scala 2 release process, it wasn't clear when you generate the zip distribution

That's handled over in https://github.com/scala/scala-dist ; the release step is "Trigger two scala-dist jobs on travis" (the first of the two jobs is the relevant one)

I wonder if automating the SDKMAN step might be overkill. We don't do a new Scala release that often — every couple months, something like that. Suppose we just added it to the release steps as a manual step, how long would that manual step take? Because if the answer is, I don't know, 5 minutes or less, something like that, automating it might not be necessary. (Automating something takes work up front. And, such automations tend to be fragile and thus may take ongoing work to maintain, too.)

[marc0der]

@SethTisue Thanks, that makes perfect sense. I think both options are super easy.

• two simple REST calls to our Vendor API for releasing the version and announcing the release
• I'll write you an SBT plugin that does all the above, much like our Gradle and Maven plugins do

I would need to share the API credentials with you over a secure channel, so contact me directly if you would like me to generate those and send them to you.

[SethTisue]

would an sbt plugin be useful to others as well?

[marc0der]

At the moment, @eed3si9n is using the API to perform his SBT releases, so he might also benefit. If we can hear back from @anatoliykmetyuk and @prolativ, they might be able to use it for Scala 3 too.

[SethTisue]

hmm, radio silence there... seems like a judgment call, happy to go along with either decision

[bjornregnell]

I think sdkman has turned out marvelous for our minority of students at Lund Univ having linux and macos and WSL on their own machines (majority are on pure windows 10). I also like how easy it is to do sdk use and just change scala version. Nice!! Thanks!

So if a streamlined automation can help make more versions available easier/faster and also release candidates that would be awesome!

[som-snytt]

I use sdkman where supported because I haven't worked out the coursier ecosystem yet. I use ~/projects/dotty/bin/scalac for trying dotty behavior. Sometimes I still accidentally type dotc.

[bjornregnell]

Yes I think sdkman is much easier than coursier, so really good that we support latest scala in sdkman. I'd like to standardize for students on install instructions and sdkman is easy to install. Only major obstacle is that it does not work on pure windows and installing cygwin/MingGW is not an option for my beginner students and WSL is still power-user-stuff with some bumps, so we also still need to break through with https://github.com/lampepfl/dotty/issues/12502 in my humble opinion as a one-klick .msi is the prevailing majority preference...

[julienrf]

Hello! Thank you @marc0der for explaining the steps to automate the release publication to SDKMAN. It seems to be a very simple step to add to our release processes, with a nice positive impact! I am happy to be in charge of updating the Scala 3 release process to perform the necessary calls to the Vendors HTTP API. I guess this would just be an additional step here.

Regarding the sbt plugin, I am not sure. I wonder if people prefer dealing with plain old shell commands? On the other hand, it is probably the simplest way to provide us with a high-level, convenient, API for publishing the release. If this is not too much work for you I would gladly try to use the sbt plugin.

[marc0der]

Thanks @julienrf, that's great news! Perhaps for starters, we can just do this with curl from your GitHub action. I can get you set up with credentials asap so we can try this out.

@SethTisue would you be able to take this forward for Scala 2? I'd be happy to help you in any way I can, and can issue you with a separate set of credentials for this purpose.

In the meanwhile, I'll see if I can roll out an SBT release plugin at some point if it's going to help you going forward. For now, here is a page with more details about how to interact with our vendors API.

[bjornregnell]

Can this mean that also Release Candidates of Scala 3 and perhaps even nightly can be invoked by sdk install and then sdk use? That would be really cool. :)

[marc0der]

Many vendors support RC builds already (for example, Gradle do this). It is entirely up to the Scala Center if they want to go down that avenue. Even if we could get the GA builds automated for starters, it would already be a big win IMO.

I would highly discourage publishing nightly builds as this will become unmanageable in no time.

[julienrf]

Can this mean that also Release Candidates of Scala 3 and perhaps even nightly

I was first considering publishing only stable releases. We might consider publishing RCs as well if there is high demand.

[SethTisue]

would you be able to take this forward for Scala 2?

Yes. (Not this week though...)

[julienrf]

Note to us: we probably want the promotion to SDKMAN (and possibly other installers) to be triggered manually, not right after the artifacts are published to Maven Central, so that we don’t repeat the issue observed at https://github.com/scala-steward-org/scala-steward/issues/1104.

[SethTisue]

@marc0der Okay, I'm finally ready to deal with this.

We plan to release Scala 2.13.8 on Monday. (Sorry, not much notice, but there will be plenty more releases after that, so it's still worth following up on this even if it doesn't happen until after 2.13.8 is out.)

I'm also in the process of updating our release steps at https://github.com/scala/scala-dev/blob/scala-dev/.github/ISSUE_TEMPLATE/release.md . Can you suggest what I should add? (As a temporary measure, I'm adding a "see issue 782" checkbox and link. See https://github.com/scala/scala-dev/pull/804)

[som-snytt]

Thanks, I am a casual user of sdkman. Actually, I put "man" in quotes.

I'm looking forward to 2.13.13, it's not a date yet, but on that theme, it feels very close to Valentine's Day, when we'll be attending the new JLo movie.

Probably you can't crank it up for a 2.13.23 release?

There is no avoiding a very special 3rd party 2.14 release, emphasis on party.

[marc0der]

Hey @SethTisue,

To perform the release on SDKMAN, you need to use our Vendor API. Two calls in particular that you should perform are Release and Announce. I also need to issue you with some credentials. I can send these through to you if you send me your public GPG key at marco@sdkman.io.

Happy to help you in any way to make this happen.

[marc0der]

@julienrf It seems like the release of Scala 3.1.1 on SDKMAN didn't work. @helpermethod was kind enough to do it manually on our side, but could you take a look on your side at what went wrong?

[SethTisue]

you send me your public GPG key at marco@sdkman.io.

@marc0der sent

[marc0der]

Credentials in your inbox @SethTisue :smile:

[julienrf]

@marc0der The script that publishes the release on SDKMAN must be run manually, it is not run automatically when the tag is pushed on Github. It seems that in our case that script was not run at all (it does not show up here https://github.com/lampepfl/dotty/actions?query=event%3Aworkflow_dispatch).

In the PR that added that script, we also added a step to the release process, to run the script at some point. However, I see in the release meta-issue that this step was absent (hence it was skipped). Maybe @Kordyjan can tell us how he created the meta-issue? Why was this step not present?

[SethTisue]

I've closed the ticket because #806 addresses the issue for Scala 2.

I tested the instructions by adding 2.12.14, which was missing for some reason. (Somebody else had already added 2.13.8.)

Thanks @marc0der for your patience and persistence on this.

@julienrf @Kordyjan I hope you won't mind if I ask that you pursue the Scala 3 side of this on another ticket, since Scala 3 isn't actionable in this repo and the Scala 3 details are different.

[Kordyjan]

To note, automatic releases of Scala 3 to SDKMAN are ready and seem to be working. We haven't used it for 3.1.1 because someone was faster and published 3.1.1 manually.