search

opensearch-project / opensearch-sdk-java

OpenSearch SDK to build and run extensions
Apache License 2.0
28 stars 58 forks source link

Where will end to end testing for extensions be performed? #634

Open peternied opened 1 year ago

peternied commented 1 year ago

For the overall Extensions effort, there should be test cases that verify functionality end to end. While many of these case can be tested in isolation integration testing required to ensure backwards compatibility / confirm breaking changes.

In order to run these scenarios changes from OpenSearch and OpenSearch SDK + Security Plugin will need to be collected and configured for the test environment. Where should this testing be performed, this repository?

peternied commented 1 year ago

If I might offer an implementation suggestion, the security test require considerable setup it might be worthwhile to see how this is done to create flexible and readable test cases, see PrivilegesEvaluatorTest.java that uses composition of the cluster state.

minalsha commented 1 year ago

@saratvemulapalli , @ryanbogan thoughts?

ryanbogan commented 1 year ago

I'm working on an integration test framework for extensions that will modify the OpenSearch gradle plugin so that both extensions and plugins can be used at the same time. How many of the security features for extensions are going to be required vs how many are going to be optional?

peternied commented 1 year ago

@ryanbogan I don't think this is a question of 'security' features to be tested, but how would we be sure that extensions are working correctly if as we update the several dependent codebases (OpenSearch / SDK / Security / JobScheduler?). While security features have a good chance to break functionality, there are many features such as job scheduler that can also be broken at integration time that should be validated.

To be cautious ever feature would have an integration test and how many scenarios are tests would be a separate exercise on a case-by-case basis.

I'm curious about that system you are thinking about, do you think it would be able to cover these broad kinds of configurations, what kind of trade-offs or limitations are you keeping in mind? Would you want to write up a proposal in this issue or a new separate issue?

dblock commented 1 year ago

As an author of an extension I'd like to be able to easily declare a test matrix that uses any build of OpenSearch to test by extension in CI/CD, aka mostly GitHub actions.

A couple of thoughts: 1) the gradle test system of OpenSearch for plugins is very powerful 2) the gradle test system of OpenSearch for plugins is a kitchen sink. Whichever way we go I would not inherit anything from OpenSearch core to avoid having to drag the kitchen sink into the development environments.