search

apache / pekko

Build highly concurrent, distributed, and resilient message-driven applications using Java/Scala
https://pekko.apache.org/
Apache License 2.0
1.21k stars 148 forks source link

Setup the release process #130

Open jrudolph opened 1 year ago

jrudolph commented 1 year ago

Apache has certain requirements for releasing a public version. We should strive for as much automation as possible. This ticket should be an overview over all steps necessary and the progress on those items. Please add items as needed.

Let's only consider public releases like RCs and GAs here (but not snapshots).

(Please add your name and/or PRs and issues to the items as needed)

References:

Organizational steps

Maven-related steps

Apache-related steps

mdedetrich commented 1 year ago

I believe one thing that we have to do under Apache-related steps is that we have to copy the sources (not the generated binaries, i.e. jars in our case) to an SVN repo. In Apache lingo I think this is called a "source package", see https://www.apache.org/legal/release-policy.html#source-packages. The sources also need to be signed.

In our case it would just be the contents of the git repo.

jrudolph commented 1 year ago

I believe one thing that we have to do under Apache-related steps is that we have to copy the sources (not the generated binaries, i.e. jars in our case) to an SVN repo.

Just creating the source archive shouldn't be a big problem.

One of the main questions is if we can trust GHA enough to make releases there and also sign those releases (it probably makes sense to do both on one machine). Also, even if Apache considers binaries only a convenience, the reality is that > 99% of all users will use the binaries from Maven Central which after all carry a much bigger risk than the distributed source files. If we are not allowed to release from GHA or decide against it, we would have to setup something that would allow to run safe and reproducible releases from the release manager's machine (e.g. using docker).

mdedetrich commented 1 year ago

One of the main questions is if we can trust GHA enough to make releases there and also sign those releases (it probably makes sense to do both on one machine). Also, even if Apache considers binaries only a convenience, the reality is that > 99% of all users will use the binaries from Maven Central which after all carry a much bigger risk than the distributed source files. If we are not allowed to release from GHA or decide against it, we would have to setup something that would allow to run safe and reproducible releases from the release manager's machine (e.g. using docker).

If by release you are talking about a Maven release, I don't think there is any issue in GHA doing this. Many other Apache projects do this, and I think that everything aside from signing has already been setup with the https://github.com/apache/incubator-pekko/pull/129. For context, in order to get the snapshot deploys working I had to make an INFRA ticket to get credentials added as github secrets and in general I have seen that Apache is facilitating doing as much as possible via GHA but judging from the fact that Nexus username/passwords are being stored as secrets it would be surprising if keys are treated any differently.

What I predict could be annoying is the official Apache release (even though 99% of users won't use it). I have spoken to some people involved Apache and apparently the proper way to do this is to manually sign it on a machine with a key that is supposed to be stored externally (i.e. via usb) and then upload it. Such info can be outdated though (and it has been in the past)

pjfanning commented 1 year ago

I have #78 and #85 open already. Do we need to consolidate these issues?

mdedetrich commented 1 year ago

I have #78 and #85 open already. Do we need to consolidate these issues?

I would say that this issue can work as a general epic meta issue where we can track the other related issues. What would be handy is if we can update the original checklist and reference these specific issues. @jrudolph Do you want to do this? I can also just edit your post.

jrudolph commented 1 year ago

Do you want to do this? I can also just edit your post.

Please edit yourself, everyone. :)

jrudolph commented 1 year ago

What I predict could be annoying is the official Apache release (even though 99% of users won't use it). I have spoken to some people involved Apache and apparently the proper way to do this is to manually sign it on a machine with a key that is supposed to be stored externally (i.e. via usb) and then upload it. Such info can be outdated though (and it has been in the past)

Of course, we can just follow the rules as given but let's at least note how paradox the situation is: we would go to great lengths to sign source code securely which no one will use or look at (and which might be signed much more easily e.g. by signing the release tag in git). On the other hand, the binaries which everyone will be running directly and which will be much harder to verify will be released on third-party machines in a process which can much more easy be convinced to tamper with the binaries or leak the secrets...

mdedetrich commented 1 year ago

Of course, we can just follow the rules as given but let's at least note how paradox the situation is: we would go to great lengths to sign source code securely which no one will use or look at (and which might be signed much more easily e.g. by signing the release tag in git). On the other hand, the binaries which everyone will be running directly and which will be much harder to verify will be released on third-party machines in a process which can much more easy be convinced to tamper with the binaries or leak the secrets...

Oh definitely the irony is not lost on me whatsoever especially considering that Pekko is a library and not an application.

justinmclean commented 1 year ago

HI,

Users need to pointed to the offical releases hosted by the ASF. This may help [1][2][3] and of historical interest. [4]

Kind Regards, Justin

  1. https://infra.apache.org/release-distribution#channels
  2. https://infra.apache.org/release-distribution#public-distribution 3, https://infra.apache.org/release-distribution#download-links
  3. https://cwiki.apache.org/confluence/display/INCUBATOR/Distribution+Guidelines
justinmclean commented 1 year ago

Hi,

I should point out why that ASF does this is that it provides you with legal protection and means you are covered by the insurance the ASF has. Go outside these boundaries and you may not have that legal protection.

Kind Regards, Justin

mdedetrich commented 1 year ago

I should point out why that ASF does this is that it provides you with legal protection and means you are covered by the insurance the ASF has. Go outside these boundaries and you may not have that legal protection.

@justinmclean As you stated (and I suspected) such policies are likely in place due to legal reasons but as @jrudolph said, especially in the case of Pekko and its modules there is an extremely strong disconnect behind the policy and what happens in reality/practice 99% of the time (I can confirm that for the users of Pekko, almost no one is going to download/test the raw source package, they will add it as a dependency to their build tool that will be resolved via Apache's Nexus repo and if they are going to get the source its going to be via git on github).

Of course we are going to follow this rule, this isn't up to debate however is there a general avenue where this can be discussed/raised?

justinmclean commented 1 year ago

Hi,

In that case the project might have some work to do to change the user perceptions on where they obtain the software from. Even if they obtain it from elsewhere it must be based on an official ASF release.

Kind Regards, Justin

On 28 Jan 2023, at 9:45 pm, Matthew de Detrich @.***> wrote:

I should point out why that ASF does this is that it provides you with legal protection and means you are covered by the insurance the ASF has. Go outside these boundaries and you may not have that legal protection.

@justinmclean https://github.com/justinmclean As you stated (and I suspected) such policies are likely in place due to legal reasons but as @jrudolph https://github.com/jrudolph stated, especially in the case of Pekko and its modules there is an extremely strong disconnect behind the policy and what happens in reality/practice 99% of the time (I can confirm that for the users of Pekko, almost no one is going to download/test the raw source package, they will add it as a dependency to their build tool that will be resolved via Apache's Nexus repo).

Of course we are going to follow this rule, this isn't up to debate however is there a general avenue where this can be discussed/raised?

— Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pekko/issues/130#issuecomment-1407371210, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABDI6BX3IMNU5ZHQXWWTWDWUT2E5ANCNFSM6AAAAAAUHNYHDE. You are receiving this because you were mentioned.

mdedetrich commented 1 year ago

Even if they obtain it from elsewhere it must be based on an official ASF release

If "based on" includes users downloading generated JVM artifacts from the same source package as the official ASF release (which will also be the same as the git repo at the same checksum of the tagged release) then like almost every other Apache JVM project that publishes JVM artifacts then yes that will be the case.

I think the point being raised is that the Apache software package particularly for libraries that are using git is practically ceremonial/checkboxing. As pointed out earlier, pushing a signed git tag to signify a release (which then triggers a pipeline to upload artifacts to repositories generated from that exact source code for that release) technically achieves the exactly same goal, especially with github repo's being synced with gitbox.

justinmclean commented 1 year ago

Hi,

Releases are not based off "pushing a signed git tag”, releases need to manually voted on by the (P)PMC and placed in the offical ASF distribution area.. Please read the links I posted earlier.

Justin

mdedetrich commented 1 year ago

Releases are not based off "pushing a signed git tag”, releases need to manually voted on by the (P)PMC and placed in the offical ASF distribution area.. Please read the links I posted earlier.

I am aware that releases need to be voted on by (P)PMC, I am talking about the steps after a release is voted on (which does currently require placing software in the Apache Distribution area).

To clarify, I am talking about hypothetical alternative for distribution after a (P)PMC vote but I don't think this thread is a productive area for this conversation so I will leave it

pjfanning commented 1 year ago

We will need something that essentially splits the release into 2 parts.

With the Nexus part of release, we can release to Nexus staging and then after the vote, we can abandon the staged release or complete its release to Maven Central using the Nexus Repository Manager.

sbt plugins like sbt-release, sbt-ci-release, sbt-sonatype, etc. can be configured not to complete the releases - just to put them in staging.

With the source and binary distributions, there are repositories where the files can be shared. If and when the release is approved, they can be uploaded to https://dlcdn.apache.org

mdedetrich commented 1 year ago

We will need something that essentially splits the release into 2 parts.

  • prep candidate artifacts and put them in a staging area where they can be reviewed
  • if the vote on the release fails, the candidate artifacts are removed - if the vote succeeds, the the candidate artifacts are released from staging

With the Nexus part of release, we can release to Nexus staging and then after the vote, we can abandon the staged release or complete its release to Maven Central using the Nexus Repository Manager.

sbt plugins like sbt-release, sbt-ci-release, sbt-sonatype, etc. can be configured not to complete the releases - just to put them in staging.

If this is the process that we use (and its a perfectly reasonable one) then assuming we only want to push a git tag for an actual release that has been successfully voted by (P)PMC on it becomes more complicated about which of these plugins to use and in what order.

For example one way of doing things would be to just use sbt +publish to put the artifacts into staging when a (P)PMC voting round is initiated for a release, and then if it is successful then in addition creating the Apache source package we would push a git tag which would trigger sbt-ci-release to close (i.e. promote to release) the staging repository.

If using sbt +publish is not enough then in this instance we can use sbt-release to add some more steps, doing this is however somewhat misleading because its not really a release but a pre-release. Alternatively and if allowed, instead of using sbt-ci-release to promote the staging repository one can configure sbt-release to do the proper Apache Software Distribution release after a successfully (P)PMC vote , i.e. the (P)PMC member would initiate on their machine a sbt release and if done this way sbt-release would also be responsible for pushing the git tag as well as promoting the staging repository (from a previous sbt +publish).

This alternative method I think is cleanest and probably closest to the "Apache way", i.e. sbt +publish once a release process is initiated and then sbt release after a successful vote which will then handle everything behind the scenes. Even the naming of the various sbt commands is very clear in conveying the intent of whats going on.

The main issue I foresee with these method/s is that we would likely have to resort to having a static value for the version of the project in build.sbt rather than the current setup of getting the version from a git tag because the git tag won't exist yet when promoting to staging. The problem/s with these method/s can be avoided by pushing the git tag when a release process is initiated however we run into the problem where if a release fails then we have to remove the git tag otherwise its not in sync with the official Apache releases.

pjfanning commented 1 year ago

ASF projects tend to use a concept of a release manager - an actual person, who can do some documented manual steps.

We can start with having a few manual steps and automate more later.

It's more important to define a process than to tailor a process to the way that a particular sbt plugin works.

The artifacts that are voted on should be signed and we need to provide a KEYS file with the public key parts of any keys that have ever been used to sign our artifacts. From an sbt perspective, that means we need sbt publishSigned.

These keys are typically keys associated with actual people so signing the artifacts is more likely to be done on the release manager's computer than to be automated. I don't see any mechanism by which the release manager's signing key can be made available to a Github Action workflow.

mdedetrich commented 1 year ago

ASF projects tend to use a concept of a release manager - an actual person, who can do some documented manual steps.

We can start with having a few manual steps and automate more later.

It's more important to define a process than to tailor a process to the way that particular sbt plugin works.

This is why I prefer sbt-release because it is manual, i.e. it has to be manually triggered on a machine by the release manager (i.e. via sbt release or just release if you are already in the shell) to go through the documented manual steps and as part of the process sbt-release can also interactively ask for signing keys on the release managers machine (amongst other things).

sbt-release also has some nice quality of life features, for example if your current git status is unclean (i.e. you have unstaged/committed changes) it will immediately halt the release process. I think that some of these preconditions can also be configured.

The issue with sbt-ci-release is that its triggered by git tag pushes and not manually, which means in addition to not being interactive (meaning its quite limited in what we do) if we want to the git tags to be in sync with actual approved Apache releases then sbt-ci-release could only be used to promote a staging repo to release which is kind of overkill, it would be better to just use sonatypeBundleRelease from sbt-sonatype directly.

The artifacts that are voted should be signed and we need to provide a KEYS file with the public key parts of any keys that have ever been used to sign our artifacts. From an sbt perspective, that means we need sbt publishSigned.

These keys are typically keys associated with actual people so signing the artifacts is more likely to be done on the release manager's computer than to be automated. I don't see any mechanism by which the release manager's signing key can be made available to a Github Action workflow.

I already started asking these questions in #asfinfra.

mdedetrich commented 1 year ago

The artifacts that are voted should be signed and we need to provide a KEYS file with the public key parts of any keys that have ever been used to sign our artifacts. From an sbt perspective, that means we need sbt publishSigned.

These keys are typically keys associated with actual people so signing the artifacts is more likely to be done on the release manager's computer than to be automated. I don't see any mechanism by which the release manager's signing key can be made available to a Github Action workflow.

Adding this as another comment as you edited yours, but incase it's not clear sbt-release works by defining a set of already existing steps so nothing is stopped it from calling +publishSigned and provide keys as you describe. Its documented at https://github.com/sbt/sbt-release#no-git-and-no-toy-projects on how to do this, the linked example shows how to configure it when git isn't available (which will be our case since release process expects to download source directly, not from git).

The more important point that we may be missing is that sbt-pgp (which is where +publishSigned comes from) may not expect signing keys in the same way as as is documented on ASF so I wouldn't be surprised if we have to tailor some steps due to this reason (and perhaps others)

jrudolph commented 1 year ago

It's more important to define a process than to tailor a process to the way that a particular sbt plugin works.

:+1: Yep, figuring out how we want and need to do the process is the most important part, we can always get the tooling to do what we want afterwards.

Note, how in the happy case (release worked, positive vote), the procedure is not so much different from what we had for akka-http: https://github.com/apache/incubator-pekko-http/blob/main/scripts/release-train-issue-template.md#cutting-the-release. Here we also only automated until staging and then had some manual testing steps and a manual triggering of promotion to Maven Central.

The question is how to deal with the unhappy cases where something goes wrong with a release or the vote fails. A few alternatives come to mind:

In the past, I have been usually quite pragmatic about it. Before a release had been announced, I was ready to just redo the tag in the git repo and restart the whole process after a fix (usually, mutating tags or main branch will only lead to short term hassles if done in a timely fashion). On the other hand, sometimes enough of a release had already slipped (e.g. to Maven Central) so that a new release version was necessary, in which case the process was just redone.

In general, the most principled approached would be just to skip version numbers in case of a problem. That would also have the benefit of reusing most of the past processes.

jrudolph commented 1 year ago

We should make sure that we also help people giving there approving vote according to https://www.apache.org/legal/release-policy.html#release-approval which require

Before casting +1 binding votes, individuals are REQUIRED to download all signed source code packages onto their own hardware, verify that they meet all requirements of ASF policy on releases as described below, validate all cryptographic signatures, compile as provided, and test the result on their own platform.

Especially, we should clarify/understand what it means to "test the result on their own platform".

mdedetrich commented 1 year ago

We should make sure that we also help people giving there approving vote according to https://www.apache.org/legal/release-policy.html#release-approval which require

Before casting +1 binding votes, individuals are REQUIRED to download all signed source code packages onto their own hardware, verify that they meet all requirements of ASF policy on releases as described below, validate all cryptographic signatures, compile as provided, and test the result on their own platform.

Especially, we should clarify/understand what it means to "test the result on their own platform".

On the same note, one thing I want to explore is that if we end up using sbt-release, adding steps to the release process (initiated by sbt release) which would print out messages reminding the release manager on the external steps required.

There are some things we can automate, i.e. compiling and running the test suite (which can be done with sbt compile and sbt test and more complex tests can also be integrated in this way) but for other things such as announcing a release, sbt-release can just print a message saying "Did you announce the release according to https://www.apache.org/legal/release-policy.html#release-announcements y/n" as a helpful reminder for the release manager on what steps need to be done.

Note that the context behind the suggestions that I am making is two things

Note that regarding the release process, I have helped out a couple of times for Kafka release and there are lot of colleagues on my team who are Apache committers/PPMC members for various Apache TLP's so I am continuously speaking with them to get a general idea of how the release process works with other projects because while there is a the strict policy at https://www.apache.org/legal/release-policy.html#release-approval, there is some level of bespoke tailoring depending on the TLP project as long as it conforms to the ASF release process.

In the past, I have been usually quite pragmatic about it. Before a release had been announced, I was ready to just redo the tag in the git repo and restart the whole process after a fix (usually, mutating tags or main branch will only lead to short term hassles if done in a timely fashion). On the other hand, sometimes enough of a release had already slipped (e.g. to Maven Central) so that a new release version was necessary, in which case the process was just redone.

I think that for now what need to decide on is how we approach git tags because this will have an effect on how we design the release process, i.e. do we tag immediately when a release vote is started and remove it later if a release vote fails or do suspend the creation of the git tag only when a formal Apache release vote is approved? I have a personal process for suspending the creation of the git tag because its simpler in the exception case but its not a hill that I will die on if we go with other options.

pjfanning commented 1 year ago
mdedetrich commented 1 year ago
  • we should not git tag candidate builds as if they are full releases

Is this because it breaks some part of a fundamental ASF process or because it complicates what you mention aftewards (in the sense that if we do git tag release candidates than the last release candidate would need to support multiple git tags if we promote it to release)?

  • so we can tag the candidate as v1.0.0-rc1
  • when the vote passes for v1.0.0-rc1, we can add tag v1.0.0 to that same commit.
  • or we can avoid tagging the commit until we agree the release - if we think the approach above is too noisy

My concern here with using RC's as a backdrop for making a release is that a project can make many release candidates before doing an actual release thats voted on, in which case because of the process it can be a bit unclear which release candidate counter can turn into an actual release while its happening.

We would also have to mandate that we always have to a release candidate just before the release which in some cases can be excessive, i.e. think of the case where we have some rc-x that has been fully tested and is likely going to be the last rc before a proper voted release but someone makes some minor non breaking changes afterwards, i.e. some basic documentation is added after rc-x. With this process we would have to make another rc-(x+1) and go through the entire hoopla of getting everyone to test rc-(x+1) even though its kind of pointless because nothing of worth has changed.

Even if we communicate that for that specific rc-(x+1) does not need to be tested because it only has minor documentation changes, it then becomes unclear (at least to me the primary goal of release candidates is to encourage the community to do manual testing on that rc throughout the lifecycle of that release to try and weed out any bugs/concerns).

On a similar note, putting git tags onto release candidates can also help communicate the stage of the lifecycle of the release, i.e. generally speaking during RC's you don't want to merge major changes into the project (that happens after a release) and with git tags its quite clear if the project is in the stage of a release from a git log perspective (you just need to see whether the last git tag is an release candidate).

Claudenw commented 1 year ago

Take a look at https://cwiki.apache.org/confluence/display/JENA/Release+Process. This is the release process for Apache Jena. It is a much simpler project and is strictly java based, However, if you look at what it does with respect to the git repository you can see the creation of tags, and roll back on failure and other automated steps. I recommend that Pekko adopt something similar and that it also be written out in the wiki.

The verification is to verify that the build of the system matches the result built by the release. IMHO if you want to use GHA to build the release you could, but it will be a complex script. The verification of the build will still have to be done on other accounts, etc.

jrudolph commented 1 year ago

when the vote passes for v1.0.0-rc1, we can add tag v1.0.0 to that same commit.

That would mean that we would still have to rebuild because the version number is also included in the binaries. Is that what you mean? The source distribution could probably stay he same (Can it? The sources might also contain the version...), but the binaries would have to be rebuilt.

While the ASF requires the sources distribution to be validated before voting, we also need to make sure that the staged binaries are valid, so that process could still fail after a positive vote (if it had to be rebuilt for a new version number)...

pjfanning commented 1 year ago

Ultimately, we can define the version in the sbt file, we don't necessarily need to derive it from git tags.

This might better suit an ASF compliant release process.

Claudenw commented 1 year ago

This is the process from Cassandra: https://cassandra.apache.org/_/development/release_process.html

mdedetrich commented 1 year ago

Ultimately, we can define the version in the sbt file, we don't necessarily need to derive it from git tags.

This might better suit an ASF compliant release process.

This was also on my mind however the use of git to determine versions makes perfect sense for snapshots which is the current behaviour we have and I would like to keep (also the behaviour where if you have an unclean git status then a timestamp is added, handy if we end up implementing the "make a snapshot for an in progress PR" that I was talking about elsewhere).

The reason why I was bringing this up is afaik at least sbt-ci-release (and maybe other plugins/tools) rely on NOT having a strict version defined in the sbt build because rather its relies on sbt-dyn-ver to determine the version of the project. Setting strict versions in sbt would also mean having to update every pekko project and come up with a new method (i.e. its a decent amount of work).

Unless I am missing something, from whats being discussed so far the simplest solution with git tags seems to be

@Claudenw thanks for the release process of other TLP projects, will read through them to see if I miss anything and/or get some inspiration.

jrudolph commented 1 year ago

Ultimately, we can define the version in the sbt file, we don't necessarily need to derive it from git tags.

This might better suit an ASF compliant release process.

Yes, I'm starting to accept this (and that's how it has been for many projects including akka in the past). If we put it into the build file and check it in, we cannot reuse the same commit for an RC and a final build. So, I think we shouldn't directly promote an RC to a final release but will have to use the real staged release as the basis to vote on (the process based on published RCs does not seem to be mentioned in the Apache docs or in the existing examples).

So, the approx. proposed process for a final would be:

Something like that?

mdedetrich commented 1 year ago

So, the approx. proposed process for a final would be:

  • decide to release
  • create a commit which fixes the version to the target version
  • start the staging process (on the release manager's computer or GHA?) to prepare all artifacts
  • start the vote

  • if vote is successful

    • add git tag
    • promote Apache release artifacts
    • promote to Maven Central
  • if release or vote is unsuccessful, revert version fixing

Something like that?

We commented at the same time and we both came to roughly the same solution however such a solution doesn't actually need a fixed version in the build. Unless I am missing something, we only need strict versions in the sbt build if there is a misalignment between git tags and the version of the software and this only occurs if we decide to create git tag immediately when a release process is started. If however if we do git tag release candidates (as I described in https://github.com/apache/incubator-pekko/issues/130#issuecomment-1410210094) and we push create git tags only when there is a successful vote then I don't see why setting a strict build version in sbt is necessary.

pjfanning commented 1 year ago

Ultimately, we can define the version in the sbt file, we don't necessarily need to derive it from git tags. This might better suit an ASF compliant release process.

Yes, I'm starting to accept this (and that's how it has been for many projects including akka in the past). If we put it into the build file and check it in, we cannot reuse the same commit for an RC and a final build. So, I think we shouldn't directly promote an RC to a final release but will have to use the real staged release as the basis to vote on (the process based on published RCs does not seem to be mentioned in the Apache docs or in the existing examples).

So, the approx. proposed process for a final would be:

  • decide to release
  • create a commit which fixes the version to the target version
  • start the staging process (on the release manager's computer or GHA?) to prepare all artifacts
  • start the vote

  • if vote is successful

    • add git tag
    • promote Apache release artifacts
    • promote to Maven Central
  • if release or vote is unsuccessful, revert version fixing

Something like that?

Seems like a good starting point. We can always evolve the release process and hopefully simplify it over time.

The first release could require a number of candidates because it is not just the Pekko PPMC that must approve it, the Incubator PMC has to approve it too and and Incubator members may not look at Pekko builds until we have release candidates to look at.

jrudolph commented 1 year ago

Btw. I don't think a tool like sbt-release will be a big help because it cannot keep state during a complex interactive release process. That's why I suggested to use a release train issue as a checklist (which has it's shortcomings) but at least provides some way of keeping track of what's going on (and also some audit by keeping track of who checked which boxes).

mdedetrich commented 1 year ago

Btw. I don't think a tool like sbt-release will be a big help because it cannot keep state during a complex interactive release process. That's why I suggested to use a release train issue as a checklist (which has it's shortcomings) but at least provides some way of keeping track of what's going on

Honestly I was thinking of prototyping this separately, it is possible for it to keep state between steps but it would involve temporary files (which sounds hacky but is also not something unique to sbt-release, even core sbt has similar issues i.e. see https://github.com/apache/incubator-pekko/issues/107).

(and also some audit by keeping track of who checked which boxes).

If sbt-release solution is ever successful, things like this would be outside the bounds of sbt-release. The only things that sbt-release could/would do in such a case is print a message that says "Did you check if other (P)PMC members have done x, y/n". I wouldn't be surprised if the only things that sbt-release could automate would be compile/test/integration test as well doing signed artifact releases asking for signing/private keys and the other steps are just printed messages as interactive yes/no prompts making sure person is properly following the ASF release guidelines.

As an aside, can someone point out if the solution described at https://github.com/apache/incubator-pekko/issues/130#issuecomment-1410210094 is problematic? Reason I am pointing this out is that if there aren't any issues with this solution then we don't really need to do any changes aside from describing the process and getting sbt-pgp to work. We can keep all of the current "version from git tag" infrastructure and it would work as we want it to. When we get to a point of having our first release candidate, then someone would just push a 1.0.0-RC-1 git tag and that would be it (of course in addition we would have to document the process)

If there are issues with my prescribed solution than we have to plan do some more work and hence plan this.

jrudolph commented 1 year ago

If however if we do git tag release candidates (as I described in #130 (comment)) and we push create git tags only when there is a successful vote then I don't see why setting a strict build version in sbt is necessary.

That sounds as if the release manager would manually run the build and keep the workspace open for the multiple days until the process is done. Is that what you are suggesting? (Of course, you can always also just recreate the tag on the same commit later, so maybe it's not an issue.)

Is it already clear that we cannot prepare artifacts for a release automatically?

jrudolph commented 1 year ago

As an aside, can someone point out if the solution described at #130 (comment) is problematic?

It's fine for me to use a local git tag to set the version from my side and indeed easier than adapting what we already have.

mdedetrich commented 1 year ago

That sounds as if the release manager would manually run the build and keep the workspace open for the multiple days until the process is done. Is that what you are suggesting? (Of course, you can always also just recreate the tag on the same commit later, so maybe it's not an issue.)

If you are talking about an actual release then yes this is the idea. I mean the release manager would start a formal release vote and at the same time publish a 1.0.0 release artifact into staging. Others would then start voting (typically takes days) as well as testing (by downloading that staging artifact) and doing all of the other necessary steps. During this whole process then there is no git tag pushed yet, its only when the necessary amount of votes have been cast and the process is successful then the release manager would push the git tag for the actual release (i.e. "v1.0.0") which would trigger (via GHA) promoting the already existing staging artifact/s for 1.0.0 into release. In addition to this they would also create the software package and publish it (as is formally required by ASF).

It's fine for me to use a local git tag to set the version from my side and indeed easier than adapting what we already have.

Actually I was thinking that when a decision is made to create a release candidate (RC) that the git tag would be pushed onto git, i.e. v1.0.0-RC1. This would trigger publishing a JVM artifact into Apache nexus release repository which people can then resolve and test. In addition since there is a v1.0.0-RC1 tag if people want to checkout the source manually at that time to do some additional testing its very easy to do so with git tags.

This makes it also ultra clear that what the release candidates are (i.e. the release candidates in the nexus repo is in perfect sync with git tags). If an RC fails or more RC's are needed, thats easy would just create a new RC and increment the RC version.

jrudolph commented 1 year ago

and getting sbt-pgp to work

Not sure what you mean with this work? Right now private keys are provided Base64 encoded via environment variable (to make it work on CI). So, the Maven part will just work.

If this is about creating the source release for the official Apache release page, then we still need to script that whole process, right? We could do that in sbt but it might be easier to throw together a bash script for that (e.g. just using git archive to create the archive and plain gpg tools to sign) + whatever is need to upload to svn.

mdedetrich commented 1 year ago

Not sure what you mean with this work? Right now private keys are provided Base64 encoded via environment variable (to make it work on CI). So, the Maven part will just work.

Are they? I asked in asfinfra and didn't get a response, could be just reading problems that don't exist. Main reason I posed this question is afaik sbt-pgp is built around gpg and I haven't set up sbt-pgp to accept keys in other formats/ways.

If this is about creating the source release for the official Apache release page, then we still need to script that whole process, right? We could do that in sbt but it might be easier to throw together a bash script for that (e.g. just using git archive to create the archive and plain gpg tools to sign) + whatever is need to upload to svn.

EDIT: I was talking about JVM artifacts published to Apache's nexus repo, these also should be signed.

jrudolph commented 1 year ago

Are they? I asked in asfinfra and didn't get a response, could be just reading problems that don't exist. Main reason I posed this question is afaik sbt-pgp is built around gpg and I haven't set up sbt-pgp to accept keys in other formats/ways.

Maybe there's a misunderstanding here? Where are other keys than gpg keys used?

I'm currently assuming we are running on the release manager's machine, not on CI if that's the confusion.

mdedetrich commented 1 year ago

Are they? I asked in asfinfra and didn't get a response, could be just reading problems that don't exist. Main reason I posed this question is afaik sbt-pgp is built around gpg and I haven't set up sbt-pgp to accept keys in other formats/ways.

Maybe there's a misunderstanding here? Where are other keys than gpg keys used?

I'm currently assuming we are running on the release manager's machine, not on CI if that's the confusion.

So according to https://infra.apache.org/release-signing.html, if I read correctly the signing keys are not stored in GPG but should be stored as separate files. This is because for security reasons, keys should not be stored on your local machine (which is typically connected to the internet) but externally on a device that is not connected to the internet (i.e. a usb stick).

Also when I was talking about sbt-pgp, I was referring to JVM artifacts and not the source package, I think a different mechanism for the source package is in order. My question here is I actually don't know whether the JVM artifacts need to use the same signing key as the source package (wouldn't surprise me if that was the case), thats what I asked in https://the-asf.slack.com/archives/CBX4TSBQ8/p1674900528637179

jrudolph commented 1 year ago

EDIT: I was talking about JVM artifacts published to Apache's nexus repo, these also should be signed.

Maven Central only requires signing and that the public keys used for signing is available in a public key repo. AFAIK there are no additional Apache requirements on Maven artifact signatures than what Maven does. In effect, these Maven signatures are worthless because there is no common process or practice to validate the signatures against anything (e.g. there's no standard that would say to look up valid keys in the KEYS file or for users to list trusted keys somewhere).

In any case, I do not think that it even matters, because either we can automate the full release in which case we must release with keys known to CI and can use them both for Maven binaries and for the sources, or we cannot automate and must provide the keys on the local machine, in which case they can also be used for both signing tasks.

mdedetrich commented 1 year ago

If thats the case then as you said its kind of arbitrary and doesn't matter, as you would just create your own private key and publish it to some key repo. I just wanted to confirm with ASF if thats the case or whether we should use the same key as the source package for Maven releases (which does provide an actual benefit, as you can securely confirm that a Maven release artifact is signed with the same release managers key that was used to sign the Apache's official source package).

Since ASF considers maven (and other release channels like docker) as convenience packages it wouldn't surprise me if the process is "do whatever you want" since its a convenience package. I guess it could be nice to use the same signing key behind the source package for the Maven artifact for reasons described earlier but I can see how that would be considered excessive.

jrudolph commented 1 year ago

If thats the case then as you said its kind of arbitrary and doesn't matter, as you would just create your own private key and publish it to some key repo.

I've heard that seriously suggested before... (not here).

I just wanted to confirm with ASF if thats the case or whether we should use the same key as the source package for Maven releases (which does provide an actual benefit, as you can securely confirm that a Maven release artifact is signed with the same release managers key that was used to sign the Apache's official source package).

Yes, I think we should.

It shouldn't be a problem because the whole release should be done on the same machine so the same single key should be available.

jrudolph commented 1 year ago

So according to infra.apache.org/release-signing.html, if I read correctly the signing keys are not stored in GPG but should be stored as separate files. This is because for security reasons, keys should not be stored on your local machine (which is typically connected to the internet) but externally on a device that is not connected to the internet (i.e. a usb stick).

This seems a bit excessive and probably infeasible because it would mean that you have to connect and disconnect the computer in the middle of the process to keep your keys safe. If you care about getting private keys stolen, a more reasonable measure would be to require a key that you have only on a hardware stick like a Yubikey. Having keys stolen in that way actually seems like a minor threat compared to supply chain attacks, attacks against the infrastructure, or against the release machines (or CI) directly.

mdedetrich commented 1 year ago

Yes, I think we should.

Agreed

It shouldn't be a problem because the whole release should be done on the same machine so the same single key should be available.

Well the only technical problem is the one I mentioned before, strictly speaking if we follow ASF's guidelines than that same key that is used for source packages needs to be stored as a file (i.e. NOT gpg since it stores the key on the local machine) which means that sbt-pgp would need to accept keys from a file (usually from a mounted usb?).

I don't know if that is possible with how sbt-gpg is currently designed

This seems a bit excessive and probably infeasible because it would mean that you have to connect and disconnect the computer in the middle of the process to keep your keys safe. If you care about getting private keys stolen, a more reasonable measure would be to require a key that you have only on a hardware stick like a Yubikey. Having keys stolen in that way actually seems like a minor threat compared to supply chain attacks, attacks against the infrastructure, or against the release machines (or CI) directly.

I believe those instructions are deliberately general to allow for things like yubikey (which I was personally thinking of). Regarding disconnecting/reconnecting your computer, that is true if follow the reasoning to its natural conclusion although I would suspect that the primary reason behind these rules is to just not leave the keys stored on the machine typically connected to the internet which is a much larger security attack vector than only having them visible on the internet at the exact time the release manager is doing a release and I agree with this sentiment.

justinmclean commented 1 year ago

Hi,

A couple of things to keep in mind about ASF releases:

Kind Regards, Justin

mdedetrich commented 1 year ago
  • You need to vote on a source release, and this needs to be a signed tar or zip archive. You can’t vote on a git tag, as tags can be changed.

Yes this is known, if it wasn't clear before all voting is done on signed tar/zips and a git tag is a convenience measure that is only done after a successful vote

  • You can’t make changes to the release after it has been voted on. If you make changes, you’d need to have another vote.

Good to know, will have to come up with another method

  • Any anther distributions need to contain what is in the voted-on source release with no modifications or extra bits.

This is also clear, the question/problem we have is whether there is a problem with the release manager signing that distribution method (i.e. signing JVM library jar's) with the same signing key that is used for the source package (tar/zip)

jrudolph commented 1 year ago
  • You can’t make changes to the release after it has been voted on. If you make changes, you’d need to have another vote.

Good to know, will have to come up with another method

What in the proposed process would require changing the sources after the fact?

This is also clear, the question/problem we have is whether there is a problem with the release manager signing that distribution method (i.e. signing JVM library jar's) with the same signing key that is used for the source package (tar/zip)

What should be the problem here?