Release TestNg more frequently

[martinaldrin]

Hi,

I see a need of release TestNg more frequently, Today it seems to be 6-9 month between the releases. And the number of features included in TestNg is kind of big, In our project we do releases for every commit, which allow fast feed back.
And in our project we have restriction on using forks.

The risk I see now is if we find a bug in a release we need to wait very long time until next release, and then there is a risk that something else have changed.

Maybe a release every quarter wouldn't create to much extra job for your team.

What do you think?

[krmahadevan]

We dont have release branches because at any point in time we would like to target for only 1 release. What that release would be (Major/minor/bugfix) is something that we decide at that point in time.

I believe we are following https://trunkbaseddevelopment.com/release-from-trunk/ I have seen other projects such as Selenium project also follow the release-from-trunk approach and not create release branches.

We have just the master branch from which we release at any given point in time. Post that, the master branch gets tagged. I am personally not in favour of making the release process complicated with introducing release branches which gets cherry-picked etc., because its going to take time in getting it done.

The branch-for-release I think would work when you want to deploy an artifact into an environment which gets executed (for e.g., a microservice that gets deployed into an environment).

For libraries I think its fine to follow release-from-trunk

[Elisedlund-ericsson]

Considering there is a CVE fixed on master branch would it make sense to release a 7.6.2 (or 7.7.0) https://nvd.nist.gov/vuln/detail/CVE-2022-4065

I surprised such a serious commit did not immediate new release.

What happened with more frequent releases? Now there's huge amount of commits piled up and even if you do a release now, we risk that there is some other change that will prevent us from uplifting to the new release forcing us to either:

1. Live with or workaround the new issues
2. Keep using old TestNG version with the CVE
3. Having to create a fork with on the old release with the CVE patch released

[krmahadevan]

@Elisedlund-ericsson - Can you please help vet out this below release candidate and let me know if all is well at your end for version 7.7.0 ?

If you are using Maven then please add

<repository>
    <id>maven-central-staging</id>
    <url>https://oss.sonatype.org/content/repositories/orgtestng-1084</url>
</repository>

to your pom file.

If you are on Gradle please use

repositories {
    mavenCentral()

    maven {
        url 'https://oss.sonatype.org/content/repositories/orgtestng-1084' }
}

Once I hear back from you and if all is well then I will publish this release candidate.

[krmahadevan]

What happened with more frequent releases?

Because I was out of action for a couple of months owing to a medical emergency !

[krmahadevan]

Now there's huge amount of commits piled up and even if you do a release now, we risk that there is some other change that will prevent us from uplifting to the new release forcing us to either:

Do you have suggestions on how to make it better?

TestNG does not have a a lot of contributors. Majority of the time, it's only me who is fixing bugs etc., So that means that the number of fixes that lands in a version is very miniscule and it takes sometime before a sizeable number of issues are fixed which would warrant a release. That explains the lag in the release.

[Elisedlund-ericsson]

Can you please help vet out this below release candidate

Yes we can forsure test it, We even tested latest TestNG master mid November already. (which I guess went fine)

Because I was out of action for a couple of months owing to a medical emergency !

No worries, Prioritize health/family over work! Thanks for the great efforts,

Do you have suggestions on how to make it better?

TestNG does not have a a lot of contributors. Majority of the time, it's only me who is fixing bugs etc., So that means that the number of fixes that lands in a version is very miniscule and it takes sometime before a sizeable number of issues are fixed which would warrant a release. That explains the lag in the release.

How I usually do it and how I prefer is when every commit gets released. then automatically uplifted and used. that way if there's a problem you run into it very quick and also it narrows it down to just a one or a few commits that could have caused the issue, and the person that did the change usually problem area still left in his recent memory.

basically test/use everything as early as possible and fail as fast as possible.

I'm fine with a release containing a major backwards compatibility issues sometimes so it can be identified as early as possible to then either be fixed/discussed or handled at the user side(migrated)

Doing it like above however means that the release process has to be fully automated. Even preferably with acceptance testing from end users included. Like us (or other big users like OpenJDK(i think?))

Maybe a more sane or reasonable first step is to make a new release like ever ~3-5 small commits or every time there is a big/important commit. of course we already test a SNAPSHOT version this often, but we rather want use a release in our production this often

[krmahadevan]

Yes we can forsure test it, We even tested latest TestNG master mid November already. (which I guess went fine)

I will wait to hear back from you before I publish it to Maven central and wrap up the rest of the release formalities.

[krmahadevan]

How I usually do it and how I prefer is when every commit gets released. then automatically uplifted and used. that way if there's a problem you run into it very quick and also it narrows it down to just a one or a few commits that could have caused the issue, and the person that did the change usually problem area still left in his recent memory.

Do you have an OSS project sample that does releases this way, which I can refer to ?

Doing it like above however means that the release process has to be fully automated. Even preferably with acceptance testing from end users included. Like us (or other big users like OpenJDK(i think?))

The release process is already automated. Just that the triggering part is manual. Do you have a bunch of acceptance tests that you can share, which we can try to incorporate and build some sort of an automated way of getting this done ?

Basically for every SNAPSHOT publishing that goes into Maven Central snapshot repository, this project's build job would run having it consume the SNAPSHOT. TestNG build snapshot would be the upstream job and the acceptance test would be the downstream job.

of course we already test a SNAPSHOT version this often, but we rather want use a release in our production this often

If you can share a bunch of acceptance tests, that can be used as a vetting out process, we can perhaps have it setup to be built after every merge to master, which publishes a snapshot version. That way, for every commit the acceptance tests are already checking if all is well.

[Elisedlund-ericsson]

Do you have an OSS project sample that does releases this way, which I can refer to ?

Everything I know of is closed source unfortunately. but basically its like your current process except that there is no manual triggering instead it runs the release flow on every commit, Then consumers(users) has CI to auto uplift to the new release and run tests, given any failures it would then report back to the origin project.

If you can share a bunch of acceptance tests, that can be used as a vetting out process, we can perhaps have it setup to be built after every merge to master, which publishes a snapshot version. That way, for every commit the acceptance tests are already checking if all is well.

Probably not that much that we can share (as in code) :( and if we have something general enough it could as well just be contributed unit/integration tests merged to TestNG itself, stuff like: https://github.com/cbeust/testng/pull/2712/files#diff-72c2d0dc194a0c36f76620fbc687e4627fa52d80619f22ce400c6116ce809126 )

(Our testing is more in the nature of uplifting latest TestNG and running with all our know working "product" tests to see if something that worked previously instead now fails)

The final results of our integration/acceptance testing of TestNG can probably be shared however in some automated way. Like posting an Issue in Github that SNAPSHOT X causes compatibility failures, then of course we would have to make an analysis of our internal closed logs / tc fails to pin point the exact issue and provide stacktraces.

[krmahadevan]

but basically its like your current process except that there is no manual triggering instead it runs the release flow on every commit, Then consumers(users) has CI to auto uplift to the new release and run tests, given any failures it would then report back to the origin project.

Are you suggesting that we release a version to Maven Central for every commit ? I don't think I quite understand this part, because our release process essentially releases the artifact into Maven Central (We first publish release candidates into staging area, which would wait there to be released or dropped). If we drop it from a staging area it means that we are discarding that release candidate, if we release it, then it means it gets published to Maven Central. @juherr any idea on this ?

The straight forward way that I can think of is that, you folks have a dedicated job internally that consumes TestNG snapshot dependency and runs your evaluations on it and report back any breakages back to us on the snapshot so that it can be fixed. I dont see a different way of doing this, if there are no acceptance tests that can be shared.

[krmahadevan]

@Elisedlund-ericsson - Just to ensure that we aren't losing focus, please do ping back the results of your evaluation on the release candidate so that I can close the release, do the post release activities and send out release notes to the TestNG users community. I will wait to hear back from you on that.

[juherr]

I don't see any advantage to make a release on every commit. Here, we are not building a huge product that suffers from the synchronization of teams.

If you really want a release from every commit, you can use https://jitpack.io as a maven repository or you can make your own release process on a synchronized fork you manage.

[vlsi]

I guess the key issue for now (which is different from the initial comment in this "issue") is that it would be great to have TestNG releases shortly after CVE is fixed.

---

On top of that, it would probably make sense to do patch releases (security-only) for several past versions of TestNG since people will be under pressure of "fixing CVE", and they might not have enough time to upgrade to the latest TestNG as CVE arrives.

For pgjdbc, we settled on back-porting security fix to all past versions till 42.2.x (42.2.0 has been released ~5 years ago, and 42.2.x is the latest version that still supports Java 1.6 and 1.7). For instance, a recent https://github.com/pgjdbc/pgjdbc/security/advisories/GHSA-562r-vg33-8x8h has been fixed in four versions 42.2.27, 42.3.8, 42.4.3, 42.5.1, even though, the versions are most likely compatible. The users would most likely upgrade from 42.2 to 42.5 without any issues, however, we decouple "urgent upgrade needed for CVE resolution" from "upgrade for fixing bugs/adding features".

One of the reasons we do patch releases for past versions is that Spring Boot has a policy that they do not upgrade across "major" versions no matter what, so if a certain Spring Boot version uses pgjdbc 42.3.x, then that Spring Boot will use 42.3.x only. See https://github.com/pgjdbc/pgjdbc/issues/2599

[bobshie]

Hello, I see there is a candidate version 7.7.0-rc2 7.7.0-rc1, where is the candidate release path, I can get the JAR and make some verification.

is /org/testng/testng/7.7.0-SNAPSHOT/testng-7.7.0-20221206.103750-3.jar the latest one?

[krmahadevan]

@bobshie - Did you read through the comments ? Please see here https://github.com/cbeust/testng/issues/2665#issuecomment-1339138206

[bobshie]

@krmahadevan , sorry, I didn't go though all comments here. OK, I'll do some verification in this week on the version: https://oss.sonatype.org/content/repositories/orgtestng-1084/org/testng/testng/7.7.0/

[bradh]

I added staging and bumped to 7.7.0 (we were previously on 7.6.1), and did a full rebuild.

No problems noted.

Our use of testng is pretty basic, and certainly wouldn't have tested every capability. It is probably reasonably representative of a lot of other projects though.

[krmahadevan]

@Elisedlund-ericsson - Is there an ETA by when you would be able to tell me the result?

[krmahadevan]

I guess the key issue for now (which is different from the initial comment in this "issue") is that it would be great to have TestNG releases shortly after CVE is fixed.

On top of that, it would probably make sense to do patch releases (security-only) for several past versions of TestNG since people will be under pressure of "fixing CVE", and they might not have enough time to upgrade to the latest TestNG as CVE arrives.

For pgjdbc, we settled on back-porting security fix to all past versions till 42.2.x (42.2.0 has been released ~5 years ago, and 42.2.x is the latest version that still supports Java 1.6 and 1.7). For instance, a recent GHSA-562r-vg33-8x8h has been fixed in four versions 42.2.27, 42.3.8, 42.4.3, 42.5.1, even though, the versions are most likely compatible. The users would most likely upgrade from 42.2 to 42.5 without any issues, however, we decouple "urgent upgrade needed for CVE resolution" from "upgrade for fixing bugs/adding features".

One of the reasons we do patch releases for past versions is that Spring Boot has a policy that they do not upgrade across "major" versions no matter what, so if a certain Spring Boot version uses pgjdbc 42.3.x, then that Spring Boot will use 42.3.x only. See pgjdbc/pgjdbc#2599

@vlsi - Thanks for sharing those additional details in terms of back patching. I will keep this part in mind going forward. There was a PR that someone had raised to fix this issue https://github.com/cbeust/testng/pull/2806

I read an interesting comment in the same PR that basically questions the seriousness of this issue since the code path in question is related to a test jar mode of execution of TestNG. Details here

[Elisedlund-ericsson]

@Elisedlund-ericsson - Is there an ETA by when you would be able to tell me the result?

All test so far have been fine, like we tested on roughly 300+ repos(typical small OSS libraries) that does just typical unit testing.

However the more heavy stuff like function/system/integration testing we have however not finished yet which uses much more of TestNG like: every type of Listener and every annotation and many of the application parameters including -xmlPathInJar

I read an interesting comment in the same PR that basically questions the seriousness of this issue since the code path in question is related to a test jar mode of execution of TestNG. Details here

I also don't fully agree with the seriousness of the issue either, Have a very hard time seeing any real world scenarios were it can be abused. in this case its more about company policies of using software with CVE and "management" not liking to see CVE: 7.8 HIGH as it looks bad.

[krmahadevan]

@Elisedlund-ericsson -

However the more heavy stuff like function/system/integration testing we have however not finished yet which uses much more of TestNG like: every type of Listener and every annotation and many of the application parameters including -xmlPathInJar

Thanks for the inputs. Can you please tell me by when can we expect to hear back from you on the completeness of those tests as well? That way I would be able to wrap up the release part in time. Please let me know. Given the fact that there are now queries and apprehensions on the CVE, I don't want to stall the release more because now I will have to keep answering all those queries :)

[Elisedlund-ericsson]

Thanks for the inputs. Can you please tell me by when can we expect to hear back from you on the completeness of those tests as well? That way I would be able to wrap up the release part in time. Please let me know. Given the fact that there are now queries and apprehensions on the CVE, I don't want to stall the release more because now I will have to keep answering all those queries :)

I actually don't know why it taking this long either, (and I'm not running it myself personally) I asked for ETA

From my perspective it ok if you go ahead and release right now. We did a complete test during mid November so the majority of the new commits(since 7.6.1) does not cause us any issues for us atleast. From the newer commits I cannot see anything I foresee as risky except possibly the "dependsOnMethods" commits which we rely on and that has caused us issues before. on the ~smaller chance of issues/failure we could instead look into a 7.7.1 so other users do not have to wait for a 7.7.0

[krmahadevan]

@Elisedlund-ericsson - Thank you so much. I will proceed with the release.

[krmahadevan]

@Elisedlund-ericsson TestNG 7.7.0 is released.

[Elisedlund-ericsson]

TestNG 7.7.0 is released.

Great, All of our tests came back successful now also :)

Thanks for all the help 🥇