[plugins] repository-xxx: extract to separate repository

[reta]

Is your feature request related to a problem? Please describe. Right now repository-xxx plugins are part of the core and as such, follow the same release cycles as the OpenSearch itself. In such a model it is difficult to contribute bugfixes and enhancements in a timely manner.

Describe the solution you'd like It would be beneficial to extract all repository-xxx plugins (or in future, each individual plugin) into separate repository, with own release cycle.

Describe alternatives you've considered Keep things as-is but this is not sustainable

Additional context N/A

[reta]

@dblock I think it would be safe to target it to 1.3.0? I will try to pick it up shortly and get a fair understanding of the efforts, thank you

[dblock]

I think this technically breaks backwards compatibility, doesn't it? @nknize WDYT?

[reta]

I think this technically breaks backwards compatibility, doesn't it? @nknize WDYT?

It shouldn't, we will be releasing a fully compatible versions with the core, but just outside the core, but let @nknize clarify :-)

[dblock]

@dblock I think it would be safe to target it to 1.3.0? I will try to pick it up shortly and get a fair understanding of the efforts, thank you

Thanks! I think if you make a new repo for it that look like all the other plugins we can talk about moving it into the org.

Another thought, does it make sense for a single opensearch-azure plugin or collection of plugins?

[reta]

Thanks! I think if you make a new repo for it that look like all the other plugins we can talk about moving it into the org.

👍

Another thought, does it make sense for a single opensearch-azure plugin or collection of plugins?

This is a good one, I initially thought about it as Azure only, but I at later point come to conclusion it is better (arguably) to move off all the repository plugins. It is easier to maintain, no need to jump between repositories (one is in core, one is somewhere else), less surprises. Also, we could move each one to separate repository later on if there would be a need to do so.

[reta]

@dblock @nknize it seems like this is doable, I have created this development repository [1], all tests and checks are passing, if there is no objections to follow this way, we could do migration with history (I just did simple copy + push for the sake of saving time), would need your help for that.

In the essence, the following modules and fixtures where moved off:

 ── plugins
│   ├── repository-azure
│   ├── repository-gcs
│   ├── repository-hdfs
│   └── repository-s3
└── test
    └── fixtures
        ├── azure-fixture
        ├── build.gradle
        ├── gcs-fixture
        ├── hdfs-fixture
        ├── krb5kdc-fixture
        ├── minio-fixture
        └── s3-fixtu

What do you think, does it make sense? :-)

[1] https://github.com/reta/opensearch-repository-plugins

[dblock]

Before we do this ...

1. Originally, I was definitely thinking "a collection of Azure plugins" vs. "a collection of repository plugins" because anyone working with Azure would want a collection of that, it seemed. But maybe I'm wrong?
2. Would one want to bring all these options into a cluster at once? Would users prefer to pick and choose? Making 5 repositories seems like a pain, but they could then evolve on their own?
3. These are taking OpenSearch as a dependency if I am not mistaken. Will we need to release each/a collection every single time anyway? So have we just made our lives 5x more difficult until we have better extensibility?

Would love to hear from @nknize.

[reta]

Thanks @dblock

Originally, I was definitely thinking "a collection of Azure plugins" vs. "a collection of repository plugins" because anyone working with Azure would want a collection of that, it seemed. But maybe I'm wrong?

I think we have only 1 plugin for Azure ... Not sure what else would be there?

Would one want to bring all these options into a cluster at once? Would users prefer to pick and choose? Making 5 repositories seems like a pain, but they could then evolve on their own?

Correct, trade-off, there is no dependency between those, so we could go with N repos (1 per plugin) or 1 repo for all repository-xxx plugins (like the poc)

These are taking OpenSearch as a dependency if I am not mistaken. Will we need to release each/a collection every single time anyway? So have we just made our lives 5x more difficult until we have better extensibility?

Correct, the blob store APIs are in OpenSearch. The release cycle depends on the https://github.com/opensearch-project/OpenSearch/issues/1707, if we relax the version requirements, the lock step releases won't be necessary.