Release TestNg more frequently

martinaldrin commented 3 years ago

Hi,

I see a need of release TestNg more frequently, Today it seems to be 6-9 month between the releases. And the number of features included in TestNg is kind of big, In our project we do releases for every commit, which allow fast feed back.
And in our project we have restriction on using forks.

The risk I see now is if we find a bug in a release we need to wait very long time until next release, and then there is a risk that something else have changed.

Maybe a release every quarter wouldn't create to much extra job for your team.

What do you think?

krmahadevan commented 3 years ago

@martinaldrin - The frequency of a release is usually determined by what is getting shipped in a particular release. Today the rate at which issues are getting fixed is pretty slow, for lack of enough folks to help out. We would love to do more frequent releases, but we also need to ensure that we dont end up doing a release which contains a very small set of bug fixes/features.

What is the minimum number of issues which would warrant a release, this can be debated. But this primarily one of the main reasons behind elongated timelines between releases.

askfor commented 3 years ago

I think there could be some bug-fix releases between feature releases, 7.4.x for example.

cbeust commented 3 years ago

@krmahadevan I think it's perfectly fine to ship minor version releases more frequently even if they only contain a handful of bug fixes.

The way I set things up in the Maven repo and github should make it trivial for you and @juherr to deploy to Maven at any time, let's try to do that more frequently.

martinaldrin commented 3 years ago

That would be very helpful, and I think that will benefit both developers and users. You will get much faster feedback on the releases if a new issues is introduced.

It is always easier to correct bugs when the code is fresh, correcting bugs that was introduced 9 month ago can be much more tricky since the code might have changed and might be affected by other code commits.

I also believe that if you need to dig into code changes that was implemented long time ago will require more effort compare to something that was recently implemented.

krmahadevan commented 3 years ago

@krmahadevan I think it's perfectly fine to ship minor version releases more frequently even if they only contain a handful of bug fixes.

@cbeust Yes. I agree. Just that our few handle of bug fixes are also only trickling in, and not flowing in :) Hence the elongated time.

The way I set things up in the Maven repo and github should make it trivial for you and @juherr to deploy to Maven at any time, let's try to do that more frequently.

Yep. This would be the first time we are going to be doing a release after the overhaul of our gradle build. Will setup sometime for this two couple of issues which @martinaldrin is focussing on gets resolved so that he and the others can basically start using 7.5.0 (or) whatever that version is going to be.

krmahadevan commented 3 years ago

That would be very helpful, and I think that will benefit both developers and users. You will get much faster feedback on the releases if a new issues is introduced.

It is always easier to correct bugs when the code is fresh, correcting bugs that was introduced 9 month ago can be much more tricky since the code might have changed and might be affected by other code commits.

I also believe that if you need to dig into code changes that was implemented long time ago will require more effort compare to something that was recently implemented.

@martinaldrin - Trust me I would love to do more frequent releases (monthly or a quarterly). And I am not against that notion. But like I mentioned, we are short of hands on helping with bug fixes. Currently its mostly just me and @juherr and sometimes it gets difficult to time slice between OSS and your day job. That's the only reason why we have elongated releases. With support of folks like you in terms of helping out with bug fixes etc on TestNG, I am confident that we can get this sorted out pretty easily.

martinaldrin commented 3 years ago

@martinaldrin - Trust me I would love to do more frequent releases (monthly or a quarterly). And I am not against that notion. But like I mentioned, we are short of hands on helping with bug fixes. Currently its mostly just me and @juherr and sometimes it gets difficult to time slice between OSS and your day job. That's the only reason why we have elongated releases. With support of folks like you in terms of helping out with bug fixes etc on TestNG, I am confident that we can get this sorted out pretty easily.

I fully understand that, and we appreciate your excellent support

juherr commented 3 years ago

@krmahadevan Do you need any help with making a release? Feel free to ping me in case of a problem.

krmahadevan commented 3 years ago

@juherr - I will try to go ahead with doing a release once all the inflight defect fixes are merged. I will surely ping you incase I run into problems.

krmahadevan commented 2 years ago

We should now be able to do quarterly releases or more frequent bug fix releases depending upon the size of issues.

martinaldrin commented 2 years ago

Hi, We have some bug corrections submitted on master, I wonder if it is time for a 7.5.1 release now? It would be very valuable for us if we could get releases more often.

Thanks /Martin

cbeust commented 2 years ago

I think it would be healthy to have a steady train of releases (e.g. monthly).

krmahadevan commented 2 years ago

@martinaldrin - TestNG 7.6.0 is now released. Please refer to the release notes here https://groups.google.com/g/testng-users/c/BAFB1vk-kok/m/uwQlDyqkAAAJ

martinaldrin commented 2 years ago

Great, many thanks for the fast release!

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0? https://github.com/cbeust/testng/tags

The reason why the tag is important for us is because our FOSS handling requires that we point out the source of the release. and now it is a missmatch between the release and the source

krmahadevan commented 2 years ago

@martinaldrin

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0?

Yes the tag is correct.

martinaldrin commented 2 years ago

Hi @krmahadevan, then I don't understand the release procedure. Shouldn't the tag match the actual release?

krmahadevan commented 2 years ago

@martinaldrin - The gradle plugin that we are using adds that as a tag because it attempts at doing release candidates. You can refer to the documentation here

martinaldrin commented 2 years ago

Always when I see other projects releasing candidates there is a mapping between the version and the tag. But I treat 7.6.0 as final release and 7.6.0-rc1 as a candidate of 7.6.0. So I do not understand this. If you planned to release a candidate rc1, then I expect the version in maven central to be named rc1 as well and not 7.6.0

Elisedlund-ericsson commented 2 years ago

@martinaldrin

A question, the tag is named 7.6.0-rc1, is that correct and the release is 7.6.0?

Yes the tag is correct.

that does not make any sense at all, So there is no official release, its just release candidate? or is the rc1 referring to a different commit than what 7.6.0 is based on? which commit is 7.6.0 based on?

The gradle plugin that we are using adds that as a tag because it attempts at doing release candidates.

That is fine but a release candidate is not a release, its a candidate to become a release, you should probably still tag the release commit with the release number.
a release candidate and and a release can be the same commit which has two tags. were the release tag is added first when the release candidate is proven to be stable and good enough to become a offical release.

krmahadevan commented 2 years ago

@Elisedlund-ericsson

So there is no official release, its just release candidate? or is the rc1 referring to a different commit than what 7.6.0 is based on?

7.6.0 is the official release. The commit represented by the tag 7.6.0-rc1 is what 7.6.0 was built on.

martinaldrin commented 2 years ago

I totally understand the procedure, but If I look at other projects that do release candidates. they do not tag the candidates, but they always tag the release. Here is an example from a different project

https://mvnrepository.com/artifact/org.aspectj/aspectjrt https://github.com/eclipse/org.aspectj/releases

Elisedlund-ericsson commented 2 years ago

The commit represented by the tag 7.6.0-rc1 is what 7.6.0 was built on.

Why not tag it with 7.6.0 then like basically every other git project. (including every other TestNG release! even the ones that used release candidates)

krmahadevan commented 2 years ago

@Elisedlund-ericsson - Its not done yet. But would like to understand as to how would that be a blocker for you folks to consume the release. The release was done only in the morning. I am still going through the clean up activities associated with a release.

Elisedlund-ericsson commented 2 years ago

The release was done only in the morning. I am still going through the clean up activities associated with a release.

That is fine, and big thanks for the release. sorry if i have sounded pushy or negative.

But would like to understand as to how would that be a blocker for you folks to consume the release.

We can temporarily consume and test it. But we cannot register it in any FOSS handling software's we have, (for handling licenses and vulnerabilities etc) as it requires mapping between Artifact and commit/tag. that prevents us to use the release in an official manner

martinaldrin commented 2 years ago

Hi,

I have some bad news with last release, We tested a snapshot last week from master and that was working. But there seems to be a new commit in-between that affects us badly. non of the executions passed.

I expect it is related to this commit: https://github.com/cbeust/testng/commit/d190ade58a58e20ced922d47a9114958a5a05e8a

Seems that some tmp path is added infront of the path

argument: -xmlpathinjar suites/postrelease/emulator_wrat_postrelease.xml

Exception: 2022-05-18T08:44:26,315 ERROR
java.nio.file.NoSuchFileException: /tmp/testngXmlPathInJar-15086412835569336174/suites/old_action_test_suites/rnc_action_regression_with_configurationdata.xml

krmahadevan commented 2 years ago

@Elisedlund-ericsson

That is fine, and big thanks for the release. sorry if i have sounded pushy or negative.

Given the fact that I juggle around with a lot of things, apart from taking care of TestNG stuff, it would be good if you could please give it some time (at least 24 hours). I am still trying to come to terms with the Gradle plugin that we are using to do releases and understand what all it does internally. So I may be a bit rusty.

But we cannot register it in any FOSS handling software's we have, (for handling licenses and vulnerabilities etc) as it requires mapping between Artifact and commit/tag. that prevents us to use the release in an official manner

Thanks for adding this context. I have created a new tag 7.6.0 which also tracks the same commit as 7.6.0-rc1 as can be seen here

krmahadevan commented 2 years ago

Hi,

I have some bad news with last release, We tested a snapshot last week from master and that was working. But there seems to be a new commit in-between that affects us badly. non of the executions passed.

I expect it is related to this commit: d190ade

Seems that some tmp path is added infront of the path

argument: -xmlpathinjar suites/postrelease/emulator_wrat_postrelease.xml

Exception: 2022-05-18T08:44:26,315 ERROR java.nio.file.NoSuchFileException: /tmp/testngXmlPathInJar-15086412835569336174/suites/old_action_test_suites/rnc_action_regression_with_configurationdata.xml

Can you please log this as a new issue ?

martinaldrin commented 2 years ago

yes, I will

juherr commented 2 years ago

@vlsi Any reason why the release step doesn't add a release tag?

vlsi commented 2 years ago

Well, ./gradlew -Prc=1 publishDist should have pushed the tag. Did it create one in your local git clone? Do you have gradle logs by chance?

krmahadevan commented 2 years ago

@vlsi - Just to summarise, I performed a TestNG release by running this workflow manually and providing the Release candidate value as 1

./gradlew -PghGitSourceUsername=cbeust -PghGitSourcePassword=${{ secrets.GITHUB_TOKEN }} -PghDryRun -PghNexusUsername=${{ secrets.SONATYPE_USER }} -PghNexusPassword=${{ secrets.SONATYPE_PASSWORD }} -Psigning.secretKeyRingFile=$HOME/.gnupg/secring.gpg -Psigning.password="" -Psigning.keyId=${{ secrets.GPG_KEY_ID }} -Prc=${{ github.event.inputs.release_candidate }} prepareVote

This published the jar to Nexus and also added the git tag 7.6.0-rc1

Questions:

Is the gradle task correct or should we be using ./gradlew -Prc=1 publishDist (as you mentioned in your comment). The documentation doesn't mention the task publishDist so I wasn't aware of it and didn't try it out as well.
If we are to be using ./gradlew -Prc=1 publishDist what value should we be providing for the -Prc parameter (I mean should there be a release candidate that exists or is just a number that we provide similar to what we do in prepareVote)
Does the task publishDist also take care of pushing the tag ? Your comment suggests that it should but still thought I would explicitly ask.
- If yes, does it add the rc to it ? I ask this because the artifact version doesn't have the -rc in it but the git tag is having that. I noticed this when using prepareVote and this basically creates an out of sync situation between the version published in Maven Central and the git tag. And to fix that, I had to explicitly create a tag that matches the version and push it to the repository.
- If the tag created adds the rc to it, how do we avoid it (i.e., the tag should have just the version,but not the rc number)

It would be great if you could please help take sometime and help out with these questions

vlsi commented 2 years ago

I performed a TestNG release by running this workflow manually and providing the Release candidate value as 1

Ok, I see. The workflow assumes the user would "close and release" the staged repository manually.

There might be multiple staged repositories, so I assume that "push of the tag" should be accompanied by "releasing the staged repository".

doesn't mention the task publishDist so I wasn't aware of it and didn't try it out as well

Oh, that was a typo in the documentation.

what value should we be providing for the -Prc parameter (I mean should there be a release candidate that exists or is just a number that we provide similar to what we do in prepareVote)

Exactly. You should provide a release candidate number matching one of the previous prepareVote tasks.

The meaning of -Prc=1 or -Prc=2 is like "I'm preparing/releasing a release candidate 1 or 2".

prepareVote prepares a release candidate, that is why it adds -rc to the tag name. publishDist takes the release candidates, and it releases a version without -rc.

krmahadevan commented 2 years ago

@vlsi - Thank you so much for adding those additional details. This clarifies a lot of things for me and has helped get a better understanding.

prepareVote prepares a release candidate, that is why it adds -rc to the tag name.

But this doesn't add -RC1, -RC2 and so on to the pom version. So we have a situation wherein the jar gets published without the RC suffixes and the git tag ends up getting published with the -rc suffixes.

Is it possible for us to have prepareVote publish the jars also with the -RC suffixes ? That way, TestNG can push release candidates into Maven Central (versions with -rc suffixes) and when we feel we are ready for a release, we can run a new workflow (I guess this needs to be created newly now based on the additional context you added, wherein we invoke publishDist with the -Prc parameter, wherein we provide a suitable value to the artifact. ) to do the actual release. I am guessing that this would also additionally create a git tag that does not have the -rc suffix in it ?

Please let me know on this.

vlsi commented 2 years ago

But this doesn't add -RC1, -RC2 and so on to the pom version.

I might be wrong here, however, my intention is 1) Go for "binary equivalence" of the release candidate and the release 2) Go for reproducible archives (building the same thing should yield the same byte-by-byte result)

In other words, if a release candidate is "good", then it should be fine to just use it as a release without building it again. For instance, if you add -rc... versions to the POM, manifests, etc, then version parses might treat it differently, and it might be that they yield different behavior for rc vs non-rc versions.

For instance, if I see a release candidate to have SHA of 2fab483..., then I would expect that the final release should have the same checksum.

That way, TestNG can push release candidates into Maven Central (versions with -rc suffixes) and when we feel we are ready for a release

There might be multiple options here:

a) You could publish -SNAPSHOT versions for each commit. Then users would be able to just specify X.Y-SNAPSHOT and check the most recent version.

b) prepareVote task does close the repository in Nexus. In other words, it forbids adding new contents to the repository, and it provides a public URL so everybody can review the content. You could share that "staging repository URL" so users could add that instead of Maven Central, and they could check the upcoming release candidate. Here's an example. Here's a release candidate for Apache JMeter: https://lists.apache.org/thread/3bqbp50ob279womj55jtpzo3tvk7t6mk. Here's the corresponding "staging repository" https://repository.apache.org/content/repositories/orgapachejmeter-1077/org/apache/jmeter/

You could add https://repository.apache.org/content/repositories/orgapachejmeter-1077 to your Maven or Gradle list of repositories, and it would resolve JMeter 5.5 artifacts from there.

If you want to use "release candidates" as a milestone release, then I would suggest updating the version in the first place. In other words, some systems use "-milestone" or "-alpha" or "-beta" release versions. You can have that if you like. Nothing really prevents from having a version like TestNG 8.0-milestone1 (see https://octopus.com/blog/maven-versioning-explained and https://github.com/gradle/gradle/issues/13050#issuecomment-625281093), and you could just publish versions and tags like that.

If you still think it is worth using -rc suffix for versions in POM when building with -Prc, then you could probably get that by adding -rcN to project.version if rc project property is detected.

krmahadevan commented 2 years ago

@vlsi - I need some help. Based on all the discussions we had, I have attempted at releasing a hotfix for TestNG 7.6.1. Below are steps that I have followed.

I ran Publish to maven central workflow to create the Release candidate 1. This ran to completion successfully and just like you had suggested, it went through with the following information which could be consumed via maven as well.

TestNG v7.6.1-rc1 is ready for preview.
Git SHA: 2312fca5476d8c9681d54127cea6618762333b91
Staging repository: https://oss.sonatype.org/content/repositories/orgtestng-1082

Now I am trying to finish the release for the above mentioned bugfix and for that I created a new workflow called release-maven-central.yml which essentially uses the command

./gradlew -PghGitSourceUsername=cbeust \
-PghGitSourcePassword=${{ secrets.GITHUB_TOKEN }} \
-PghDryRun \
-PghNexusUsername=${{ secrets.SONATYPE_USER }} \
-PghNexusPassword=${{ secrets.SONATYPE_PASSWORD }} \
-Psigning.secretKeyRingFile=$HOME/.gnupg/secring.gpg \
-Psigning.password="" \
-Psigning.keyId=${{ secrets.GPG_KEY_ID }} \
-Prc=${{ github.event.inputs.release_candidate }} \
publishDist

But when I ran this workflow, I am seeing an error which says

Build testng-root FAILURE reason:                                
    Caused by: Tag 7.6.1-rc1 is not found. Please ensure you are using the existing release candidate index for publishing a release
        at com.github.vlsi.gradle.release.StageVoteReleasePlugin$configureRoot$11$2.run(StageVoteReleasePlugin.kt:289)
        at com.github.vlsi.gradle.release.StageVoteReleasePlugin.runValidations(StageVoteReleasePlugin.kt:392)
        at com.github.vlsi.gradle.release.StageVoteReleasePlugin.access$runValidations(StageVoteReleasePlugin.kt:63)
        at com.github.vlsi.gradle.release.StageVoteReleasePlugin$configureRoot$11.execute(StageVoteReleasePlugin.kt:314)
        at com.github.vlsi.gradle.release.StageVoteReleasePlugin$configureRoot$11.execute(StageVoteReleasePlugin.kt:63)

The complete run logs can be found here

The tag 7.6.1-rc1 is actually there in github.

I am not sure what is missing here. Can you please help ?

cc @juherr

vlsi commented 2 years ago

Caused by: Tag 7.6.1-rc1 is not found.

It looks like checkout action did not fetch the tag from the remote repository, and the release plugin does not attempt to fetch tag from the server.

Here's the log: https://github.com/cbeust/testng/runs/7114674266?check_suite_focus=true#step:2:461

   * [new ref]         2312fca5476d8c9681d54127cea6618762333b91 -> origin/master

It looks like depth: 0 in actions@v3 would fetch all the tags: https://github.com/actions/checkout#fetch-all-history-for-all-tags-and-branches

An alternative option is to add an extra "attempt to fetch release tag from the repository in case it is missing" to the release plugin.

krmahadevan commented 2 years ago

@vlsi - I moved over to using actions@v3 and updated the workflow and re-ran. It now fails with the below error

Caused by: org.gradle.api.internal.provider.AbstractProperty$PropertyQueryException: Failed to query the value of extension 'nexusStaging' property 'stagingRepositoryId'.

Here's the run log for reference.

vlsi commented 2 years ago

That is indeed the limitation: currently release plugin uses a file on the local system to store stagingRepositoryId.

I'm afraid that is still not implemented in gradle-nexus/publish-plugin yet, see https://github.com/gradle-nexus/publish-plugin/issues/19#issuecomment-775496387.

The workaround would be creating the file with repository id manually: create build/stagingRepositories/nexus.txt that contains stagingRepositoryId

krmahadevan commented 2 years ago

@vlsi - Thank you so much for responding back so quickly.

The workaround would be creating the file with repository id manually: create build/stagingRepositories/nexus.txt that contains stagingRepositoryId

Can you please help me with this?

Quick googling says I can use an action such as this one to create the file. Or do you recommend any other specific actions that I can use.
Where would I get the stagingRepositoryId. Is it orgtestng-1082 (or) is it 1082 that I need to extract from the output of publish-maven-central workflow which is Staging repository: https://oss.sonatype.org/content/repositories/orgtestng-1082

vlsi commented 2 years ago

Where would I get the stagingRepositoryId. Is it orgtestng-1082 (or) is it 1082 that I need to extract from the output

That is indeed the ID you need.

You could grab build/stagingRepositories/nexus.txt after https://github.com/cbeust/testng/blob/master/.github/workflows/publish-maven-central.yml#L40. However, I am not sure how are you going to pass the ID to the release workflow.

So I would suggest adding a required input parameter to Release TestNG to Maven Central workflow, so it asks for "staging repository id" (e.g. orgtestng-1082), and then you could put something like cat $STAGING_REPOSITORY_ID > build/stagingRepositories/nexus.txt before https://github.com/cbeust/testng/blob/master/.github/workflows/release-maven-central.yml#L42

krmahadevan commented 2 years ago

@vlsi

I moved further along. I decided to get the repository id as a user input parameter.

But now I am hitting this error

Build testng-root FAILURE reason:                                
    Execution failed for task ':releaseRepository':
        io.codearte.gradle.nexus.infra.NexusHttpResponseException: 500: Server Error, body: [errors:[[id:*, msg:Unhandled: Missing staging repository: orgtestng-[108](https://github.com/cbeust/testng/runs/7130620262?check_suite_focus=true#step:6:109)2

Complete run logs are available here

Not sure what am I missing :(

vlsi commented 2 years ago

@krmahadevan , would you please try using echo -n ... > ..? It looks like there's a newline after the repository id.

krmahadevan commented 2 years ago

@vlsi - Thanks for that hint. It finally went through https://repo1.maven.org/maven2/org/testng/testng/7.6.1/

Now with respect to the tag, I am a bit confused. Should I create a tag called 7.6.1 and have it mapped to the same commit id as that pointed to by https://github.com/cbeust/testng/releases/tag/7.6.1-rc1 because the binaries are going to be the same, although I did push commits in but which were mainly related only to the github workflow files and so has no effect on the binary ?

krmahadevan commented 2 years ago

@juherr @cbeust @missedone - Based on all the inputs that @vlsi shared and based on the release that I did for 7.6.1 I have documented the release process in this wiki page. Please help take a look.

missedone commented 2 years ago

@krmahadevan thanks for spending so much time/effort on articulating the release process and implementing it with GHA hence ideally anybody in the testng team can do the release. i definitely look forward more frequent release and would like to align the process on the testng eclipse plugin and testng remote projects as well

bobshie commented 2 years ago

hello @krmahadevan , do you have the next plan about testng new version released? can we release beta TestNG quarterly for us to verify some fixed change.

Thanks.

krmahadevan commented 2 years ago

@bobshie - Sure. Infact you can already consume the snapshot version of TestNG (https://oss.sonatype.org/content/repositories/snapshots)

I would like to go through the issues and see if we can include a few more bug fixes into it before we go ahead and release.

askfor commented 2 years ago

I think there should be bug fix releases, 7.6.1 for example. That means the branching model should change as well, not just releasing process.

krmahadevan commented 2 years ago

I think there should be bug fix releases, 7.6.1 for example. That means the branching model should change as well, not just releasing process.

We wouldn't be needing a branching model because we aren't running parallel releases. Its trunk based development model. So the pom file gets updated with whatever version. For e.g.,

if its a bugfix release only z gets changed in x.y.z
if its a minor release we change y in x.y.z and
if its a major release we change x

askfor commented 2 years ago

If use TBD model, maybe there should be some release branches, then it's possible to cherry pick bug fixes and release a bug fix version? https://trunkbaseddevelopment.com/branch-for-release/

testng-team / testng

Release TestNg more frequently #2665