search

apache / lucene

Apache Lucene open-source search software
https://lucene.apache.org/
Apache License 2.0
2.67k stars 1.03k forks source link

Update release process to work with Gradle. [LUCENE-9488] #10527

Closed asfimport closed 3 years ago

asfimport commented 4 years ago

The release process needs to reflect using Gradle rather than Ant. I suspect this will be a significant task, thus it has its own JIRA

Migrated from LUCENE-9488 by Erick Erickson (@ErickErickson), resolved Oct 13 2021 Attachments: Skjermbilde 2021-10-05 kl. 11.18.28.png Linked issues:

Sub-tasks:

Pull requests: https://github.com/apache/lucene-solr/pull/1860, https://github.com/apache/lucene-solr/pull/1860, https://github.com/apache/lucene-solr/pull/1860, https://github.com/apache/lucene-solr/pull/1860

asfimport commented 4 years ago

ASF subversion and git services (migrated from JIRA)

Commit 3134f10a4255b5d766462251868a43e9bb9c03ce in lucene-solr's branch refs/heads/master from Mike Drob https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3134f10

LUCENE-9488 Update release process to work with gradle (#1860)

asfimport commented 4 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

I don't think lucene/version.properties should be restored. This duplicates information. If such a file is needed, it should be generated (from within gradle) so that it's up-to-date.

asfimport commented 4 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

No please don't add lucene/version.properties. This file was just there for ant, so the release manager can easily change the version after release.

For Gradle, the only file you need to edit is root's build.gradle. The version number (main version) has to be inserted there. The same comment as in version.properties is also there, so the regex after release must just be updated slightly.

So please revert and remove the file again, as it will run out of date.

asfimport commented 4 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

Hi I added a review comment in this PR. You should have asked the Gradle maintainers for review :-)

asfimport commented 4 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

If such a file is needed, it should be generated (from within gradle) so that it's up-to-date.

It's the other way round: Ant consumed that file and the release process was updating the release version in it when creating new branches and so on. Modifying or creating it is completely useless, as the file is ignored and not read at all.

asfimport commented 4 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Thanks for having this under control, Uwe.

asfimport commented 4 years ago

Uwe Schindler (@uschindler) (migrated from JIRA)

The release script also uses the wrong command line parameter to set release version (multiple places in script):

This must be {{-Dversion.release=%s}}, as gradle has some reserved meaning of {{version}}.

I added this a review comment.

asfimport commented 4 years ago

Mike Drob (@madrob) (migrated from JIRA)

Thanks for the pointers, I'll address them in the next set of patches I put up for this. The release process was also consuming that file, which is why I thought I needed to put it back, but I'll figure out how we can consume the other files instead. Will tag all of you on the next review, this issue is by no means complete.

asfimport commented 4 years ago

ASF subversion and git services (migrated from JIRA)

Commit 3134f10a4255b5d766462251868a43e9bb9c03ce in lucene-solr's branch refs/heads/SOLR-14866 from Mike Drob https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3134f10

LUCENE-9488 Update release process to work with gradle (#1860)

asfimport commented 4 years ago

ASF subversion and git services (migrated from JIRA)

Commit 08e38d3452d548189edba691c2853b63b1fa55ae in lucene-solr's branch refs/heads/master from Mike Drob https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=08e38d3

LUCENE-9488 Create Release Artifacts with Gradle (#1905)

Co-authored-by: Dawid Weiss <dawid.weiss@carrotsearch.com> Co-authored-by: Uwe Schindler <uschindler@apache.org>

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Mike, I'm currently updating releaseWizard in #10848. I came across these lines in buildAndCheckRelease.py:

https://github.com/apache/lucene/blob/main/dev-tools/scripts/buildAndPushRelease.py#L120:L121

if gpgKeyID is not None:
  # TODO sign
  # cmd += ' -Psigning.keyId=%s publishSignedPublicationToMavenLocal' % gpgKeyID
  pass

Is the gradle build ready for gpg signing yet? Do you want to address it here in this jira? Over in #10848 I'm removing Solr stuff from the scripts...

asfimport commented 3 years ago

Mike Drob (@madrob) (migrated from JIRA)

I think @hossman did some work in the gpg signing, I’m not up to date on the latest unfortunately.

asfimport commented 3 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

gradle build gpg support for release signing was added in #10975

See help/gpgSigning.txt for details on what project properties & build targets are involved.

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Thanks for the pointer. I committed and tested gpg signing in #10848

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

The gradle build needs to assemble source package as well as zip and tgz. See comment in https://github.com/apache/lucene/blob/main/lucene/packaging/build.gradle#L80:L81

// The "main" distribution is the binary distribution.
// We should also add 'source' distribution at some point
// (we can't do it now as the build itself is tangled with Solr).

 

asfimport commented 3 years ago

Mike Drob (@madrob) (migrated from JIRA)

The gradle build needs to assemble source package as well as zip and tgz.

Why can’t we have release wizard do git archive?

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

git archive is an option (it doesn't check for uncommitted/ staged files but I assume it's not a problem). Another option is adding this to the build (although it'll be tricky with all the exclusion patterns required).

One more subtle issue is that (I believe) the build assumes that git structures are always available - it won't succeed on a "bare" set of sources... I'm not sure how much of a problem this is.

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

That's a good question. Should it be possible to download the source release of Lucene, unzip it and then do a new gradle assemble, without being on a git checkout. Probably not.

The structure of the 8.10 source package is identical to the lucene/ sub folder of the lucene-solr git repo. And that works for ant/ivy since it is self contained. But on main, the lucene sub folder is not self contained, the user needs some build files from the root folder in order to run gradle, including help files. If we want to start packaging the entire git repo including dev-tools, then git archive may be the easiest way out. But perhaps we should clean up the duplicated README and LICENSE files before that?

One way is perhaps to move all text files from lucene/ to top level and rename the lucene/ folder as modules/ ?

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

These moves will make merging for backports more difficult - this was the reason they stayed where they were. A simple rename of lucene/ to modules/ will also entail a fairly intense set of replaces all over the build files. I'm not against it but it will be quite involving.

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

I tried git archive:

git archive --format tgz --prefix lucene-9.0.0/ --output lucene-9.0.0-src.tgz HEAD

The output is

Skjermbilde 2021-10-05 kl. 11.18.28.png

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Yep, looks good to me (it includes everything that is versioned).

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

I could add it to the release wizard. But we also like to make checksum and gpg signatures for the source artifact, so would it perhaps be better to integrate that git archive command into the gradle build, so that the packaging/build/distributions/ folder will contain eveything, and we use the same signing method as for the binary artifacts? I suppose gradle can execute git commands..

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

It surely can. grep for quietexec - some of the source generation tasks use this. I have a feeling the entire distribution should be assembled from within gradle (as a signed configuration artifact) but I can't really dig into this now, I'm really sorry about it.

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Got it working, here's the PR: https://github.com/apache/lucene/pull/353 Room for improvements so happy for review feedback and/or commits directly into the PR branch!

asfimport commented 3 years ago

ASF subversion and git services (migrated from JIRA)

Commit 5cd0d68a06c32ea9787540a2d36db9f816ee3a2b in lucene's branch refs/heads/main from Jan Høydahl https://gitbox.apache.org/repos/asf?p=lucene.git;h=5cd0d68

LUCENE-9488 Assemble source tar, with checksum and signing (#353)

Also see #11189 for a future cleanup of this code

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

The old ant target 'prepare-release' invokes the targets 'clean, package, generate-maven-artifacts', and 'package' target includes 'package-src-tgz,create-package,documentation,-dist-changes'. It all ends up with this folder structure in lucene/dist, which is then ready to upload to staging area:

├── changes
│   ├── Changes.html
│   ├── ChangesFancyStyle.css
│   ├── ChangesFixedWidthStyle.css
│   └── ChangesSimpleStyle.css
├── lucene-8.10.0-SNAPSHOT-src.tgz
├── lucene-8.10.0-SNAPSHOT-src.tgz.asc
├── lucene-8.10.0-SNAPSHOT-src.tgz.sha512
├── lucene-8.10.0-SNAPSHOT.tgz
├── lucene-8.10.0-SNAPSHOT.tgz.asc
├── lucene-8.10.0-SNAPSHOT.tgz.sha512
├── lucene-8.10.0-SNAPSHOT.zip
├── lucene-8.10.0-SNAPSHOT.zip.asc
├── lucene-8.10.0-SNAPSHOT.zip.sha512
└── maven

In main/9.0 we execute targets 'assembleDist, signDist, mavenToLocalFolder' which ends up with this folder structure in the 'lucene/packaging/build/distributions' folder:

├── lucene-9.0.0-src.tgz
├── lucene-9.0.0-src.tgz.asc
├── lucene-9.0.0-src.tgz.sha512
├── lucene-9.0.0.tgz
├── lucene-9.0.0.tgz.asc
├── lucene-9.0.0.tgz.sha512
├── lucene-9.0.0.zip
├── lucene-9.0.0.zip.asc
└── lucene-9.0.0.zip.sha512

I.e. no 'maven' or 'changes'.

The maven files end up in 'build/maven-local', and the changes files seem to exist in 'lucene/documentation/build/site/changes'.

Question is whether the gradle build has a target to assemble these into one folder where buildAndPushRelease expects everything to be, or if we should add extra logic to buildAndPushRelease.

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

I added this "maven-local" for just verifying what's being published. It isn't really used anywhere. https://github.com/apache/lucene/blob/main/gradle/publishing/maven-local.gradle

These artifacts should also be signed - the current setup doesn't sign anything, just publishes Maven metadata (which is sufficient for local repositories, for example).

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Jan, do you know the process by which maven artifacts are "published" to maven central in 8x? Is this through sonatype nexus or some other means?

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Ok, guess that will also be part of #11189 then. I have started looking at smoketester in #11036 which validates all of this.

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

I can add the assembly of these maven files tonight. I'd still like to know how they're being published to maven central in 8x - never followed the release process (shame on me).

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

The release Wizard says this: https://github.com/apache/solr/blob/main/dev-tools/scripts/releaseWizard.yaml#L976:L1028 

So seems we first call ant task 'stage-maven-artifacts' which gets them into ASF nexus repo, then click a button there to push them to Central. PS: You should attemt the RM job some time :) 

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Right. So it's the same thing as with Sonatype's Nexus (OSS) - manual close/promotion in the user interface. I'm not sure how ant was uploading Maven artifacts but from within Gradle we can do it directly (the folder under distribution can be created too but it'll be for human inspection only - artifacts are uploaded directly from their build locations).

I should try to push a release one day, sure. The thing is - I hate doing it, even at my daily job!... :)

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Think the ant build uses Maven Ant task to do the job, so equivalent for gradle should work. Just a matter of passing in the correct repo coordinates: repository.id=apache.releases.https and  repository.url=https://repository.apache.org/service/local/staging/deploy/maven2

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Yep. We already do some of the work pushing nightly builds. I'll take a look at this and will get back to you with the required modifications to the release wizard. Sorry for not being able to do it right now.

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Hi Jan. I've added collecting changes and maven artifacts... see the commits here: https://github.com/apache/lucene/pull/359

But... it's all looking rather badly to me. I'd rather clean it up properly before we make the 9.0.0 release. I mean: it's a shiny new major version so maybe we should have the release process upgraded too? If you don't mind waiting a few days and working with me, I'll be glad to help.

Anyway, if you'd like to try out the PR, please do - gradlew assembleDist will collect everything in that 'distributions' folder (and this is wrong too... but let's leave with it for now). You may have to set up gpg signing in your \~/.gradle/gradle.properties:

signing.keyId=
signing.password=
signing.secretKeyRingFile=

If you're running with an agent, the password can be blank.

The thing I couldn't get past is Nexus. In theory, this should upload artifacts to staging:

gradlew publishJarsPublicationToApacheReleasesRepository -PasfNexusUsername= -PasfNexusPassword=

I can log in to repository.apache.org and I see the releases/staging repository but I can't upload anything:

Could not PUT 'https://repository.apache.org/content/repositories/releases/org/apache/lucene/lucene-classification/9.0.0-SNAPSHOT/maven-metadata.xml'. Received status code 400 from server: Bad Request

I'm not sure whether it's insufficient karma or something else. I looked at the ant build but I don't see anything special there... Can you try and see if you can upload anything to the staging environment via gradle?

An alternative to having gradle upload artifacts to Nexus directly would be to prepare a "nexus release bundle" (ZIP with all the artifacts and metadata inside) which you can then upload in Nexus's user interface... This would have the benefit that all the "release" binaries would indeed be prepared in a single location - binaries, sources and Maven artifacts.

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Took me two hours to figure out why Nexus returns HTTP 400... It's because the upload command above attempts to send artifacts with a snapshot release number and the release repository doesn't allow such artifacts.

It's stupid that it doesn't reply with something more informative. I'm done for the day but I think I have all the pieces – will polish this over the weekend and will get back to you, Jan.

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Congrats! No stress. I think we have some decent progress on release process now, so the job of volunteering as RM does not mean a week of fixing the tooling plus a week of doing the release :) But I hope the RM will submit a PR against dev-tools/scripts after the release is done, since there is likely to be some bumps left for when the actual real life release happens!

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

I filed a large PR with most of the packaging code rewritten from scratch: https://github.com/apache/lucene/pull/372

It does make a few changes on how things work. Please take a look at help/publishing.txt and try it out, letting me know if you run into any issues.

asfimport commented 3 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

I filed a large PR with most of the packaging code rewritten from scratch: https://github\.com/apache/lucene/pull/372

It does make a few changes on how things work. Please take a look at help/publishing.txt and try it out, letting me know if you run into any issues.

I didn't review the details of PR372, but i did try it out based on dawid's comments (somwhere?) about wether using gpg-agent still worked...

Using gpg-agent per the instructions in help/publishing.txt worked fine ... although i have no idea why it says you need --max-workers 1 ?

when i removed the explicit check dawid hardcoded for max.workers, i was able to build & sign (using my gpg-agent) the assembly using my normal 5 worker threads w/oproblem...

$ gradle -Pvalidation.git.failOnModified=false -Psign -PuseGpg -Psigning.gnupg.keyName=4DDFABAF68C0F906B76CD9A09C784577F8F58E19 assembleRelease

(maybe this --max-workers 1- check is just a different solution to the same problem already mentioned in gpgSigning.txt about Inappropriate ioctl for device, and we can/should mention --max-workers 1 as an additional workaround for that problem?)

other comments on the edits to gpgSigning.txt:

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

although i have no idea why it says you need --max-workers 1 ?

It's because of the comment in gpgSigning.txt - indeed the one about gradle being multi-threaded, and no-tty issue... I thought it'd be the simplest solution to just force gradle to be single-threaded (this is what the option above does).

and we can/should mention --max-workers 1 as an additional workaround for that problem?)

I think we should enforce max-workers the way it's currently implemented, if it's indeed a problem. People don't read txt and this will give them a clear error upon trying. On the other hand, if you've tried and it worked then... why is this needed again? :)

I will rename the gpg-agent signing file.

if folks are willing to provide their passphrase as a gradle property, they can now use -Psigning\.password=\.\.\.\. instead.  There's no reason to use -PuseGpg (and involve an external gpg / gpg-agent process)

Right... that would be the simplest solution of all. But I think some people will insist on using gpg-agent (and I understand the motives for doing this). I'd leave it in as an option - it does work, it just requires dodging some corner-cases. The release process python script should (for simplicity) ask for the password (using a secure, non-echoing tty) and pass these to gradle (via environment variables or command-line switches).

Thanks Hoss.

 

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

@janhoy Can I merge this in? Have you had a chance to take a look?

asfimport commented 3 years ago

ASF subversion and git services (migrated from JIRA)

Commit 8bcc3dc430b981dc75c7e005592848eacc745b16 in lucene's branch refs/heads/main from Dawid Weiss https://gitbox.apache.org/repos/asf?p=lucene.git;h=8bcc3dc

LUCENE-9488: rewrite distribution assembly, signing and checksum generation (#372)

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

Thanks a lot Dawid, Mike, Uwe, Hoss for this last non-trivial effort, to get a step closer to a lucene 9.0 release! 

asfimport commented 3 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

It's because of the comment in gpgSigning.txt - indeed the one about gradle being multi-threaded, and no-tty issue... I thought it'd be the simplest solution to just force gradle to be single-threaded (this is what the option above does).

...but that punishes people (like me) who have their gpg-agent configured with a "good" pinentry command (that doesn't use the tty – mine works directly with my OS keychain like a good gpg-agent should so you know you can trust it), by forcing them to only one one worker thread for the entire gradle build if they want to sign the artifacts.

Given that people (now) have to go out of their way to specify -PuseGpg the explicit failure when workers > 1 seems excessive.  Anybody who wants to explicitly use the gpg-agent should just choose to use a good pinentry command, or choose to  use  --max-workers 1 if they really want a "bad" tty based pinentry (even though that defeats the safety that gpg-agent provides because you can't tell it's really gpg-agent asking for your passphrase).

Forcing anyone that want's to sign with a gpg-agent to use  --max-workers 1 , simply because other people who don't know what they are doing might get an error if they go out of their way to -PuseGpg  seems ... bizare.

This is essentially saying "If you want to use the safest possible mechanism for gpg signing (gpg-agent + out of band passphrase), then you don't get to use any parallelization in your build –  because parallelization can't work for people who use a less safe mechanism (gpg-agent + passphrase to gradle's tty) and we don't want to make them set an extra option – even though people willing to trust gradle's tty with their passphrase already have another (non gpg-agent) option for doing the build with parallelization"

Why are we hobbling the feature designed for people protective of their passphrase just to coddle the people who aren't protective of their phrase when we've already added a simpler alternative for those very trusting people?

if folks are willing to provide their passphrase as a gradle property, they can now use {{-Psigning.password=....}} instead There's no reason to use {{-PuseGpg}} (and involve an external gpg / gpg-agent process)

Right... that would be the simplest solution of all. But I think some people will insist on using gpg-agent (and I understand the motives for doing this). I'd leave it in as an option

I'm not at all suggesting that we remove gpg-agent support – i would absolutely -1 that.

I'm only talking about removing the docs regarding -Psigning.gnupg.passphrase=... ... it existed before purely as a way to support the "I'm lazy and don't have gpg-agent configured well and i want to provide my passphrase on the command line" usecase – but we no longer need it now that you've hooked in support for -Psigning.password=....

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

I’ll try to make buildAndPushRelease.py and wizard configurable with both signing methods. But perhaps default to gpg-agent. The RM needs gpg anyway for smoke tester etc, so we cannot support a release without a proper gpg setup.

asfimport commented 3 years ago

Dawid Weiss (@dweiss) (migrated from JIRA)

Seems like I misunderstood the comments you made in that help file then, Chris. Can you rephrase what's already committed (and change the commands to what actually works for you)? I'm really not familiar with this area so I can't authoritatively state what's going to work (that's why I added te max-threads, defensively).

asfimport commented 3 years ago

Chris M. Hostetter (@hossman) (migrated from JIRA)

Seems like I misunderstood the comments you made in that help file then, Chris. Can you rephrase what's already committed (and change the commands to what actually works for you)? I'm really not familiar with this area so I can't authoritatively state what's going to work (that's why I added te max-threads, defensively).

Sure thing: #11210

asfimport commented 3 years ago

Jan Høydahl (@janhoy) (migrated from JIRA)

I filed #11211 to update buildAndPushRelease

asfimport commented 3 years ago

ASF subversion and git services (migrated from JIRA)

Commit f64c81c3f83ee00885a127fa4adc00c34583cc2c in lucene's branch refs/heads/main from Chris M. Hostetter https://gitbox.apache.org/repos/asf?p=lucene.git;h=f64c81c

11210: remove max-worker restriction added by LUCENE-9488 when 'useGpg' in effect

Also update docs to remove the point of confusion that lead to thinking that restriction was useful

asfimport commented 2 years ago

Adrien Grand (@jpountz) (migrated from JIRA)

Closing after the 9.0.0 release