Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds

dblock commented 2 years ago

Is your feature request related to a problem? Please describe

Coming from https://github.com/opensearch-project/opensearch-build/issues/20#issuecomment-932316080, publish distribution zips to maven.

Maven can be used for things other than java dependencies and is also actively used in such cases.

Describe the solution you'd like

[x] Publish plugin distribution ZIPs to Maven
[x] Publish Snapshot builds for plugin zips to Maven
[x] Get rid of checked in snapshot plugin zips

Dependencies with plugins

Describe alternatives you've considered

No response

Additional context

Some examples of others which also publish ZIPs on maven:

Search Guard: https://docs.search-guard.com/latest/search-guard-versions
Wildfly: https://repo1.maven.org/maven2/org/wildfly/wildfly-dist/24.0.1.Final/

Acceptance Criteria

All the distribution zips for each plugin should be available in maven

rursprung commented 2 years ago

to reply to @dblock's comment from the original issue:

@rursprung Another suggestion, while still leveraging an authoritative link, how about scanning https://github.com/opensearch-project/project-website/tree/main/_artifacts for opensearch-* and then into the the version description?

opensearch-1.1.0-min-linux-x64.markdown:
---
role: daemon
artifact_id: opensearch-min
version: 1.1.0
platform: linux
architecture: x64
artifact_url: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz
slug: opensearch-1.1.0-min-linux-x64
category: opensearch
type: targz
signature: https://artifacts.opensearch.org/releases/core/opensearch/1.1.0/opensearch-min-1.1.0-linux-x64.tar.gz.sig
security_warning: true
indirect: true
---

this would require the build to connect to the "outside world" every time - using maven allows us to cache things locally with a well-established process (our internal maven server has the public maven servers as remotes and caches all packages downloaded from there). if we have to scan for files that'd be a home-grown process which is then brittle and risks falling apart whenever something changes (e.g. the structure of the markdown file you linked - that's currently not a public API, if we'd scan it you'd have to treat it as such).

dblock commented 2 years ago

One more reason to do this is to (help) fix these bugs:

dblock commented 2 years ago

@rursprung if you or someone else wants to contribute this change, starting with snapshots, we'd take it given the proliferation of checked in zips; I think the work is to include the contents of plugins when publishing to maven in distribution-build.jenkinsfile

dblock commented 2 years ago

I'd like us to consider this for 1.3.0, labelled as such.

gaiksaya commented 2 years ago

Is publishing the core-plugins zips to maven in scope of this change? Today we are publishing everything for them except the zip on maven https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/discovery-ec2/1.2.4-SNAPSHOT/

dblock commented 2 years ago

Is publishing the core-plugins zips to maven in scope of this change? Today we are publishing everything for them except the zip on maven https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/discovery-ec2/1.2.4-SNAPSHOT/

I would imagine that yes.

peterzhuamazon commented 2 years ago

[Grooming] Need to get more information on the scenario that we need this from Maven. Is the maven dependency a required one for BWC tests? Or the existing cloudfront in front of S3 is enough?

@VachaShah @adnapibar

dblock commented 2 years ago

[Grooming] Need to get more information on the scenario that we need this from Maven. Is the maven dependency a required one for BWC tests? Or the existing cloudfront in front of S3 is enough?

@VachaShah @adnapibar

The issue is to publish zips to maven so that maven can be used for dependency management.

bbarani commented 2 years ago

Can we not use artifacts.opensearch.org for dependency management? BWC tests should be able to use the artifacts directly from artifacts.opensearch.org.

dblock commented 2 years ago

Can we not use artifacts.opensearch.org for dependency management? BWC tests should be able to use the artifacts directly from artifacts.opensearch.org.

It would ideally be a maven repo, which is a system with a bunch of well designed dependency management tools. We could get away with artifacts.opensearch.org, too, but we've been asked for maven explicitly in a few issues linked above. We're reinventing the wheel and using different tools for the same purpose.

jmazanec15 commented 2 years ago

@gaiksaya @peterzhuamazon I think core-plugins zips would be helpful. I want to be able to run integration tests for k-NN on Mac. The INTEG_TEST testDistibution only ships with the transport modules (ARCHIVE does not work on mac).

I need the lang-painless module because it needs to be installed on the cluster for k-NN to work. I can pick up the artifacts for lang-painless from https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/ but I am unable to install the module on the integ test cluster. I tried zipping the lang-painless artifacts up in a custom gradle task, but it still fails to install due to a missing plugin-descriptors file, which is not included in the artifacts link above.

It would be nice to be able to declare dependencies on core plugins and easily be able to install them on the integ test clusters. I am not sure if there is a better workaround to this (if so, please let me know). The TestClusters plugin already has the ability to add modules/plugins from zips.

More details in OpenSearch issue for what I have tried in particular: https://github.com/opensearch-project/OpenSearch/issues/1676#issuecomment-1071114101.

ylwu-amzn commented 2 years ago

https://github.com/opensearch-project/opensearch-build/issues/1689, as we support latest artifact URL, we will try latest link first, like https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.1/latest/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.3.1.0.zip, https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.1/latest/linux/x64/builds/opensearch/plugins/opensearch-ml-1.3.1.0.zip ,

@amitgalitz will work on removing job-scheduler local zip in AD, @jackiehanyang Can you take a look for moving local ml-commons to this latest URL in SQL plugin.

prudhvigodithi commented 2 years ago

I can see with current setup that publishes maven jars to snapshot repo has an added date suffix format example 2.0.0.0-alpha1-SNAPSHOT/common-utils-2.0.0.0-alpha1-20220331.205721-1-javadoc.jar, (https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch/2.0.0-alpha1-SNAPSHOT/) in order to upload the generated plugin zips under builds/opensearch/plugins, we can directly publish them or go with same like adding date suffix before we publish.

The publish location can be as follows:

plugin zips for snapshot type to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin//<2.0.0.0-alpha1-SNAPSHOT>/.zip

I already see this path https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin for core plugins, we can leverage this path or they can go under https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch//

The main releases plugin zips, once finalized to releases repo, once the path is finalized https://aws.oss.sonatype.org/content/repositories/releases/ @dblock @bbarani

dblock commented 2 years ago

@prudhvigodithi Leave that decision to maven. Maven adds these suffixes to allow multiple -SNAPSHOT versions to be uploaded, so for .zips I would do the exact same thing as for other artifacts such as JARs. For the released version of the artifact I imagine the suffix is dropped.

prudhvigodithi commented 2 years ago

Hey @dblock for jars the date suffix as (20220331)

   - maven/org/opensearch/plugin/percolator-client/2.0.0-alpha1-SNAPSHOT/percolator-client-2.0.0-alpha1-20220331.205538-1.jar.md5
   - maven/org/opensearch/plugin/percolator-client/2.0.0-alpha1-SNAPSHOT/percolator-client-2.0.0-alpha1-20220331.205538-1.jar.sha512
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/opensearch-scripting-painless-spi-2.0.0-alpha1-20220331.205538-1.pom
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/maven-metadata.xml.sha256
   - maven/org/opensearch/plugin/opensearch-scripting-painless-spi/2.0.0-alpha1-SNAPSHOT/opensearch-scripting-painless-spi-2.0.0-alpha1-20220331.205538-1-javadoc.jar

but for plugin zips this date is not added (currently)Example as

 opensearch-job-scheduler-2.0.0.0-alpha1-SNAPSHOT.zip
 opensearch-ml-2.0.0.0-alpha1-SNAPSHOT.zip
 opensearch-knn-2.0.0.0-alpha1-SNAPSHOT.zip

So when we upload the zips (mvn deploy )after every distribution build under snapshot repo, with each build the zip files will be overridden as they constantly generate with same suffix as -2.0.0.0-alpha1-SNAPSHOT.zip considering above example. But if we are ok to always pick the latest build zips like under 2.0.0.0-alpha1-SNAPSHOT folder, then it should be good to be like opensearch-job-scheduler-2.0.0.0-alpha1-SNAPSHOT.zip (as example).

dblock commented 2 years ago

I am not super familiar with how it works, but I would want to take a maven dependency on opensearch-ml 2.0.0.0-alpha1-SNAPSHOT and get the latest. File names seems like an implementation detail.

prudhvigodithi commented 2 years ago

Proposed Solution

Almost inclined with the same setup on how the current jars are published to maven repo, but the key difference is instead of creating a directory(as maven coordinates and then deploy) we allow maven to do it, by creating local file system as temp maven repo.

Example:

mvn install:install-file -DgroupId=org.opensearch.plugin -DartifactId=pgplugin -Dversion=0.0.1 -Dpackaging=zip -Dfile=package.zip -DlocalRepositoryPath=../loacalrepo/

With the above cli command now we have folder created as maven coordinates loacalrepo/org/opensearch/plugin/pgplugin/0.0.1

This will have pom file and zip file stored in loacalrepo/org/opensearch/plugin/pgplugin/0.0.1

Snapshot zips:

Upload the snapshot plugin zips to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/

/-SNAPSHOT/<.zip_file> Edit the plan https://github.com/prudhvigodithi/opensearch-build/blob/slack-publish-v1/jenkins/opensearch/distribution-build.jenkinsfile#L82 to upload zips to nexus maven repo settings.xml ``` nexus ${SONATYPE_USERNAME} ${SONATYPE_PASSWORD} ``` ``` mvn --settings="${mvn_settings}" deploy:deploy-file -DgeneratePom=false -DrepositoryId=nexus -DpomFile=loacalrepo/org/opensearch/plugin/pgplugin/0.0.1/pgplugin-0.0.1.pom -Dfile=loacalrepo/org/opensearch/plugin/pgplugin/0.0.1/pgplugin-0.0.1.zip -Durl=$repo_url ``` **Release zips:** The Release candidates will be signed before staging to make sure files are unaltered. settings.xml ``` nexus ${SONATYPE_USERNAME} ${SONATYPE_PASSWORD} ``` Create a staging repo `nexus-staging:rc-open` this will create a new new staging repository on the remote Nexus Upload zips from the local filesystem to a staging repository using plugin in built `deploy-staged-repository` Close the created staging repo `nexus-staging:rc-close` ``` staging_repo_id = $(mvn --settings="${mvn_settings}" \ org.sonatype.plugins:nexus-staging-maven-plugin:rc-open \ -DnexusUrl=$REPO_URL \ -DserverId=nexus \ -DstagingProfileId="${STAGING_PROFILE_ID}" \ -DstagingDescription="Staging artifacts for build ${BUILD_ID}" \ -DopenedRepositoryMessageFormat="opensearch-staging-repo-id=%s" | grep -E -o 'opensearch-staging-repo-id=.*$' | cut -d'=' -f2) ``` ``` mvn --settings="${mvn_settings}" \ org.sonatype.plugins:nexus-staging-maven-plugin:1.6.5:deploy-staged-repository \ -DnexusUrl="$REPO_URL \ -DrepositoryDirectory="${staged_repo}" -DautoReleaseAfterClose=false \ -DstagingProgressTimeoutMinutes=30 \ -DstagingProfileId="${STAGING_PROFILE_ID}" \ -DstagingRepositoryId="${staging_repo_id}" ``` ``` staged_repo=loacalrepo (file system folder) REPO_URL = "https://aws.oss.sonatype.org/" STAGING_PROFILE_ID = "${SONATYPE_STAGING_PROFILE_ID}" staging_repo_id from above mvn cli staging command ``` @dblock

rursprung commented 2 years ago

i just re-read this ticket to see whether it'll land in 2.0 and have some comments to add:

first of all: thanks for working on this! sorry for not having helped out here, somehow time is running through my fingers and i never got around to it :(
i mainly (only?) see comments about publishing plugin ZIPs to maven, but please note that my request was also explicitly for the final distributions (i.e. the ones listed for download here: https://opensearch.org/downloads.html) as well
IMHO it'd make sense to put this on the distributions roadmap: https://github.com/orgs/opensearch-project/projects/13
is the maven repository to which you plan to publish hooked up to maven central & co. or will consumers have to add yours as an explicit remote? since other opensearch artifacts can be found here i think this will be fine as long as you publish on the same maven server: https://mvnrepository.com/artifact/org.opensearch

prudhvigodithi commented 2 years ago

Hey @rursprung The target for plugins to be part of the maven is to allow download of zip's by dependency model, off course once they are in maven they can even be downloaded using HTTP/S methods to run the required tests or to install the plugin in the cluster. Distributions (the tar format files) are good to be part of maven, but I don't see a strong need to download them using dependency model, I mean by using maven coordinates (Please correct me here if i'm wrong). They just need to download using a HTTP method and install them for desired purpose.

Coming to the Roadmap, yes I agree it would make sense to add to the roadmap. Right now the plugin creation part is done that does the zip publications to maven, the target is to add this to each existing plugin repo, that follow certain guidelines (PR for document is under review) and make sure the zips are added to maven.

The maven for releases should be central maven along with nexus.

@dblock @bbarani any thoughts to add ?

rursprung commented 2 years ago

Distributions (the tar format files) are good to be part of maven, but I don't see a strong need to download them using dependency model, I mean by using maven coordinates (Please correct me here if i'm wrong). They just need to download using a HTTP method and install them for desired purpose.

as i had already mentioned in https://github.com/opensearch-project/opensearch-build/issues/20#issuecomment-915890695 and https://github.com/opensearch-project/opensearch-build/issues/20#issuecomment-916674597 (and i think also various other places) we'd actually want to pull in the opensearch distribution also via maven as it brings a ton of benefits (caching amongst them).

prudhvigodithi commented 2 years ago

from the comment, you mean the TAR packaged files? we could also ship them along with plugin zip's based on further discussion :+1: (But 1st lets target the plugin zips and we can add TAR packaged files to this META)

rursprung commented 2 years ago

from the comment, you mean the TAR packaged files?

you're right, i mean the .tar.gz files, e.g. this one here: https://artifacts.opensearch.org/releases/core/opensearch/1.3.1/opensearch-min-1.3.1-linux-x64.tar.gz once you have windows distributions i guess that would be a ZIP, though i personally don't care about that one (then again, for consistency reasons that one should also be published)

But 1st lets target the plugin zips and we can add TAR packaged files to this META

let's do that! will you edit the ticket description or should i add a comment there?

dblock commented 2 years ago

@rursprung @prudhvigodithi Looks like current issues talk about publishing "plugin zips". After @prudhvigodithi is done with that all those will get closed rather than expanding their scope. I think we need two more related issues: one to publish the min distribution (e.g. https://artifacts.opensearch.org/releases/core/opensearch/1.3.1/opensearch-min-1.3.1-linux-x64.tar.gz) to maven, and another to publish the distribution zip (e.g. https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.1/opensearch-1.3.1-linux-x64.tar.gz) to maven, in this repo. Note that there are cons to doing that because it effectively creates 2 different places where we publish signed binaries. If you agree, care to open them @rursprung?

rursprung commented 2 years ago

i was going to raise a new ticket, but then i realised again that the title of this ticket here literally is "publish distribution ZIPs to Maven", so based on that i think it should cover it? and you raised this ticket here out of our discussion over on https://github.com/opensearch-project/opensearch-build/issues/20#issuecomment-938175872

i'm fine with raising a new ticket (or even two, as you suggested), just wondering on how to word it so that we can ensure that we then are talking about the same thing and don't get a scope change again. because so far i fully expected this ticket here to contain the distributions (min & full) and the plugins just as a (very welcome & also needed!) side-effect ;)

dblock commented 2 years ago

@rursprung yeah let's not assume side effects :) you're right though it was all zips in this issue - I decided to scope this down and rename this issue to one to only focus on publishing plugin zips - care to open 2 more for the other kind of artifacts, one for each?

rursprung commented 2 years ago

done: #2068 & #2069, i hope they contain all relevant information.

PS: that was a sneaky change of the title on this ticket ;)

dblock commented 2 years ago

done: #2068 & #2069, i hope they contain all relevant information.

PS: that was a sneaky change of the title on this ticket ;)

I "clarified" and "simplified" it :) thank you!

prudhvigodithi commented 2 years ago

Hey @dblock we have now, a mechanism to publish the zips to maven repo using opensearch.pluginzip, can we close this issue ? @bbarani

dblock commented 2 years ago

This issue is about leveraging this mechanism. At least these two need to be resolved, and we need to check that we're not doing something similar elsewhere before closing this.

prudhvigodithi commented 2 years ago

Hey Starting with release 2.1.0, we now have plugin zips in maven repo https://github.com/opensearch-project/opensearch-build/issues/1916#issuecomment-1179276885

Pending items related to this issue before it can be closed: https://github.com/opensearch-project/index-management/issues/123 https://github.com/opensearch-project/cross-cluster-replication/issues/274

@dblock @bbarani

Thank you

prudhvigodithi commented 2 years ago

Closing this issue as associated tasks are marked as complete, please feel free to re-open this issue if required. :) Thank you

opensearch-project / opensearch-build