Publish plugin ZIPs to artifacts.opensearch.org

[dblock]

Is your feature request related to a problem? Please describe

Coming from https://github.com/opensearch-project/opensearch-build/issues/1750#issuecomment-1077871774 where we are asked to publish plugin ZIPs. Currently to get k-nn-1.3.0.zip I have several options:

Use the released package to locate the .zip build

wget https://artifacts.opensearch.org/releases/bundle/opensearch/1.3.0/opensearch-1.3.0-linux-x64.tar.gz

tar vfxz opensearch-1.3.0-linux-x64.tar.gz opensearch-1.3.0/manifest.yml
x opensearch-1.3.0/manifest.yml

cat opensearch-1.3.0/manifest.yml | grep knn
    location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/1362/linux/x64/builds/opensearch/plugins/opensearch-knn-1.3.0.0.zip

wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/1362/linux/x64/builds/opensearch/plugins/opensearch-knn-1.3.0.0.zip

The drawback is that we're downloading unsigned bits and there's no strong guarantees that builds would be preserved (even if nobody plans to delete them).

Download the latest build

wget https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/latest/linux/x64/builds/opensearch/plugins/opensearch-knn-1.3.0.0.zip

The drawback is that we're downloading unsigned bits, there's no strong guarantees that builds would be preserved, and you may be downloading a newer build of 1.3.0 that was not the version released (an RC is picked amongst the recent builds).

Describe the solution you'd like

1. Publish plugin ZIPs to artifacts.opensearch.org. Modify https://github.com/opensearch-project/opensearch-build/blob/main/vars/promoteArtifacts.groovy#L61 to also include plugins. Note that today we publish native plugins, e.g. https://artifacts.opensearch.org/releases/plugins/discovery-ec2/1.3.0/discovery-ec2-1.3.0.zip. Also publish plugin ZIPs such as k-nn, e.g. into https://artifacts.opensearch.org/releases/plugins/discovery-ec2/1.3.0/k-nn-1.3.0.zip. Note that we do not publish two copies, x64 vs. arm64, but we should.

2. Document the locations of the ZIPs on https://opensearch.org/docs/latest/opensearch/install/plugins/#available-plugins

Describe alternatives you've considered

This is an alternative to https://github.com/opensearch-project/opensearch-build/issues/716 where we are trying to publish the ZIPs to maven.

Additional context

No response

[jmazanec15]

I liked the maven approach in that we could add a dependency on a non-released dependency via SNAPSHOT. For instance, if a plugin depends on another plugin.

Is there a way to address this? Maybe pulling from latest the zip from ci for dependent plugins?

[dblock]

I liked the maven approach in that we could add a dependency on a non-released dependency via SNAPSHOT. For instance, if a plugin depends on another plugin.

Is there a way to address this? Maybe pulling from latest the zip from ci for dependent plugins?

I think we should/can do both since we release the binaries for core/native plugins today, and should add the other plugins too.

[prudhvigodithi]

Once published to maven, we can give maven repo download URL for plugin zips, they can still download from maven repo the required plugins or work with dependency gradle model Example maven URL's to download https://mvnrepository.com/artifact/org.opensearch/opensearch-core/1.3.0 or https://aws.oss.sonatype.org/service/local/repositories/releases/content/org/opensearch/opensearch/1.3.0/ Moving forward we can add zips as well under https://aws.oss.sonatype.org releases repos, and snapshot zips under https://aws.oss.sonatype.org snapshot repo.

[prudhvigodithi]

We can publish plugin zips snapshots to https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/opensearch/ and the main releases plugin zips, once finalized to https://aws.oss.sonatype.org/content/repositories/releases/org/opensearch/opensearch/

This way all the zips can be downloaded on required by a click, also they can be added to dependency model and can be used to build other plugins or modules

With just putting in artifacts.opensearch.org(http://artifacts.opensearch.org/), a user can download yes, but cant use them as dependency model in gradle workflow. So its better and clean to be only in https://aws.oss.sonatype.org/ Example as https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#consume-maven-artifacts-in-order @dblock

[prudhvigodithi]

Focusing on https://github.com/opensearch-project/opensearch-build/issues/716, that should solve the problem to download the zips that are part of maven repo with clickstreams (or with curl/wget).

[prudhvigodithi]

Hey @dblock, just a thought once the META issue to publish the zips to maven is resolved, do we still need to publish plugin ZIPs to artifacts.opensearch.org? @dblock @peterzhuamazon

[dblock]

@prudhvigodithi I believe that yes. They should be downloadable directly and guessing the maven location feels like abusing what maven is built for.

[prudhvigodithi]

got it @dblock, but once the zip is in maven they can also be downloaded directly via the URL, apart from using maven coordinates, if yes we need to publish to artifacts.opensearch.org, does this not cause duplicates having at multiple places?

[dblock]

I personally don't have a problem with duplicates.

[prudhvigodithi]

Hey I see the parent issues is closed, right now since the release zips and SNAPSHOT zips are part of maven (signed and published), a user can always download from maven, I dont see a need to re-upload them again to the artifacts.opensearch.org. Also the zips are part of ci.opensearch.org, where latest can be used to get the latest development version (using wget and tar), example as: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-job-scheduler-2.1.0.0.zip

MVN can be used here to fetch the zips

mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get  -DrepoUrl=https://aws.oss.sonatype.org/content/repositories/snapshots/  -Dartifact=org.opensearch.plugin:opensearch-job-scheduler:1.3.0.0-SNAPSHOT:zip

For POM file

<dependencies>
        <dependency>
            <groupId>org.opensearch.plugin</groupId>
            <artifactId>opensearch-job-scheduler</artifactId>
            <version>2.1.0.0</version>
            <packaging>zip</packaging>
        </dependency>
</dependencies>

For a gradle project

dependencies {
    classpath "org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0@zip"
    classpath "org.opensearch.plugin:opensearch-knn:2.1.0.0@zip"
}

Since we have multiple ways to get the zip, can we close this issue ? @dblock @bbarani @gaiksaya

[dblock]

@prudhvigodithi I think we still have to document this - should we add a "Distribution Artifacts" section to https://github.com/opensearch-project/opensearch-build README, with a bit more technical content than https://opensearch.org/downloads.html, and a "Maven" section with the content above?

[prudhvigodithi]

Hey @dblock I agree with you, we should be adding some info in README about the artifacts and ways to fetch and use them. @bbarani @peterzhuamazon @gaiksaya Thank you

[bbarani]

@prudhvigodithi @dblock It looks like the details for installing plugin using Maven has been already updated on the OpenSearch.org website. Do we need to add similar information in readme.txt as well in opensearch-build repo?

[dblock]

I think we should document internals of where this publishing happens.

[gaiksaya]

Closing this issue as resolved. @dblock Maybe create a new issue to add a wiki page to this repo (See https://github.com/opensearch-project/opensearch-build/wiki) with details on what all should be documented? The readme has too much information making it hard to read. Thanks!