[Multiple Datasource] Add automated tests for version decoupling

cwperks commented 3 months ago

Opening an issue to track testing for version decoupling with multiple datasources. As part of CI, we should add checks to ensure that the security-dashboards-plugin is compatible with all the versions of clusters that it is purported to be compatible with.

It will be important to establish a definition for compatibility of the security-dashboards-plugin and prior OpenSearch versions. Core features of the security-dashboards-plugin (like the features tested during release sanity testing) should work when connected to a backend datasource of prior versions.

Core features to test include:

Create a role
Map a role to a user
List the authentication backends
List the users
List the roles

The security APIs have seldomly changed in the last series of minor releases, but one API that may be incompatible is the dynamic multi-tenancy feature introduced in 2.7.

derek-ho commented 3 months ago

Easy solution: throw: https://github.com/opensearch-project/security-dashboards-plugin/blob/feature/supporting-multiple-datasources/test/jest_integration/security_entity_api.test.ts#L426 into a matrix of runners going back to whatever we support. If the run succeeds we should be compatible

scrawfor99 commented 3 months ago

[Triage] Hi @cwperks, thanks for filing this issue. It seems smart to test something we say we support like decoupled backend and frontends. It looks like Derek has listed a possible way of adding this testing as well.

DarshitChanpura commented 2 months ago

Test scenarios:

Unknowns that should be addressed prior to testing:

[ ] features introduced in later versions should be controlled via feature flag (maybe?) (for example, rest-api permissions should not be displayed for versions 2.9 or below)
[ ] identify least supported version

PS: CRUD updates mean : create, read, update and delete security config entities

For backwards compatibility:

[ ] As admin, I should be able to perform basic CRUD operations on security config entities: user, roles, permissions, role-mappings, audit logs, authentication setup.

As an admin:

[ ] should be able to delete and add DataSources
[ ] should be able to login to local and remote cluster
[ ] should not be able to assign REST API permission for older cluster to prevent being grandfathered in upon cluster upgrade
[ ] should reflect CRUD updates only on the DataSources selected
[ ] should not be able to assign permissions that doesn't exist on the dataSource (drop-down list should be controlled by some sort of version matching)

As a normal user:

[ ] should be able to login to local and remote cluster
[ ] should not allow login to remote cluster which doesn't contain the user
[ ] should not be able to view data on remote cluster if it doesn't have permission
[ ] should be able to CRUD security config entities on remote cluster which it has access to
[ ] should not be able to auto-assign REST API permissions to older clusters (not sure if this is entirely possible, but would be nice to at-least manually test)
[ ] should be able to modify/delete dataSources only with admin permissions

opensearch-project / security-dashboards-plugin