How are modifications for AnomalyDetector going to be tracked

[peternied]

For the effort in Migrate Anomaly Detector plugin to work as an Extension (#24), where is this code going to be worked on?

I would suggest tightly coupling the initial effort to this repository to make interface changes easier to propagate, but this will lead to a point and time when the ending extension will need to be extracted out. While considering options, the codebase could be kept in sync with Anomaly detection or it could drift until the official extension is created.

[peternied]

@dbwiddis You mentioned you had some ideas on how / where this could be done, so I'm assigning this to you.

[dbwiddis]

You mentioned you had some ideas on how / where this could be done

More accurately I think I said how it shouldn't be done! :-)

Documenting the options here and noting that none of them are optimal.

Background/assumption: there will be "new" code for the extension that can live anywhere until it's merged, and there will be necessary changes to the main OpenSearch branch to support this.

Option 1: All development on the main branch.

• Advantage: Always up to date with other changes in main.
• Disadvantage: Temporary/placeholder code in main that could find its way into a release and become API before it's finalized.

Option 2: Development in a separate branch on OpenSearch repo coupled with a separate repository (either within -sdk or its own).

• Advantage: Current workflow we're using for sdk so we're used to the "publishToMavenLocal" workflow and can draw dependencies on it.
• Disadvantage: The longer we keep a separate branch the bigger the risk of merge conflicts when it finally needs to be committed. Plus the more complex the code gets the harder the final merge will be to review.

Option 3: Develop on a fork.

• Advantage: everything is self contained and nothing gets committed until final PRs.
• Disavantage: development and review is fragmented and not really available for the community to look at and review, bigger risk of diverging from source. Not a good idea for anything larger than a single feature/PR.

3 is a bad idea. It's the normal triangle workflow for PRs and it should be limited to that.

We're currently doing 2, and I'm iffy about adding another layer of redirection on that. Maintaining a separate branch and keeping it updated takes effort from someone (who is actually doing it and how often?). This is a reasonable model for defined groups of code, but is not sustainable indefinitely. When are we going to merge the existing branch? Can we merge it before starting to add a bunch of new code for AD?

1 is not a good idea for "all the things" but is a good idea for finalized features.

In my ideal world with a magic wand, we'd keep "2" as a longer running model with temporary code but have a system in place to move over "finalized" bits of code into main in order to keep the commits small enough for reasonable review.

[dbwiddis]

I think the best path forward right now is:

1. Create a feature/extensions branch on the anomaly-detector plugin repo. @saratvemulapalli will do this (if not already done)
2. Simultaneously build new feature on the sdk project main branch (just for AD as it's our first extension and will probably share lots of update code fixes with main; later ones will use branches) while deprecating functionality required in the plugin on the extensions feature branch.
3. @saratvemulapalli will make sure the extensions feature branch stays in sync with main on AD. This should not be an issue as it's not being actively developed; still the responsibility has been assigned!

[minalsha]

1> is already in place: https://github.com/opensearch-project/anomaly-detection/tree/feature/extensions