[FEATURE] Resolve discrepancies between individual repo and functional repo tests

[bbarani]

Is your feature request related to a problem?

OpenSearch distribution build uses functional test repo for executing integration tests after every successful OpenSearch dashboards build. We have been getting various questions about the process to on-board to functional repo for new plugins and also the mechanism around the guard rails for making sure that tests in individual repos are in sync with the tests in functional repo. We noticed that some teams are using their own Cypress version for running repo level tests and some are re-using the one from functional repo.

What solution would you like?

• A clear documentation of the on-boarding process along with the scope, definition and usage of functional test repo for new plugins.
• Automated guard rails to make sure that the tests are in sync with individual repos once its on-boarded.
• Clear instructions to the repo owners detailing the boundary between tests present inside functional test repo vs individual repo

What alternatives have you considered?

A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[seraphjiang]

Hi @tianleh @CCongWang @kavilla @ohltyler would you triage the feature request?

If the existing documentation is not enough or not easy to access, let's fix the gap.

do we need to sync test today, i assume all tests will go here.

cc: @bbarani @dblock

[ashwin-pc]

The developer guide, contributing guide and other important information are today all inside the contributing.md file. I've opened up a similar issue in #290. There are additional docs that I want to add too but need some clarity on the final destination for these docs. Do you think somethign like this would be ideal:

./
- DEVELOPER_GUIDE.md
- CONTRIBUTING.md
- ONBOARDING.md
- docs/
  - writing_tests.md
  - generating_data.md
  - ....

[tianleh]

Updated the doc structure.

[ashwin-pc]

@bbarani does this restructure make the instructions clearer?

[bbarani]

@ashwin-pc The restructured instructions would definitely help but we should also add some automated guard rails to make sure that we have mechanism to track the differences in tests between individual repos vs ft repos.