[META] Add tests for existing jenkins jobs

[abhinavGupta16]

Is your feature request related to a problem? Please describe

Following up on the comment - https://github.com/opensearch-project/opensearch-build/issues/1444#issuecomment-1010324850

• [ ] Create a separate folder for tests for jenkins job
• [x] #1712

The existing jenkins jobs needs tests -

• [ ] docker-build job
• [ ] opnesearch-dashboards-distribution-build job
• [ ] opensearch-ruby job
• [ ] opensearch-distribution-build job
• [ ] tar-promotion job
• [ ] whitesource-scan job
• [ ] test-orchestrator job
• [ ] testsuite job
• [ ] check-for-build job
• [ ] Add a CI check to ensure all jenkins job have a test associated with it.

Describe the solution you'd like

Every job should have a test associated with it, with a regression txt file inside jenkinsjob-regression-files

Acceptance Criteria

• All the libraries should have a lib tester associated with it
• All the jenkins job should have a test associated with it that uses only lib testers for libraries
• A check is added in CI making sure every job has a test

[abhinavGupta16]

Let's create an issue for every task here when the work is begun. Thank you!

[dblock]

Let's create an issue for every task here when the work is begun. Thank you!

FWIW we have a lot of 1-line issues, I think it's totally fine to track a list like here, link to PRs as we go, then close the issue when we have 100% done

[prudhvigodithi]

The tests already exists for all shared libraries used by Jenkinsfiles, the functions are being fully tested. Closing this issue, please feel free to reopen. https://github.com/opensearch-project/opensearch-build/tree/main/tests/jenkins

[gaiksaya]

These tests for jenkinsFiles and not the libraries. We need to add tests for the existing jenkinsFile in this repo. Might need to revisit the list mentioned in description. Once the libraries are separated, we need to add tests to check the compatibility and expected assertions in these tests.

Just to explain: There are 2 types of tests that exists in build repo today.

1. Tests that test the jenkins library (vars folder)
2. Tests that actually test the jenkins pipelines. We have mixed those up in few libraries which is incorrect. Hence need this issue to rectify it.

Example: https://github.com/opensearch-project/opensearch-build/blob/main/tests/jenkins/TestCCRPerfTest.groovy#L73-L109

[jordarlu]

Hi, updating the To-Do list/table to show which Test we should be working on...

  | Jenkins job | location | Test groovy (tests/jenkins/) -- | -- | -- | -- 1 | bwc-test.jenkinsfile | opensearch-build/jenkins/opensearch-dashboards/ | TestOpenSearchDashboardsBwcTest.groovy 2 | distribution-build.jenkinsfile | opensearch-build/jenkins/opensearch-dashboards/ |   3 | integ-test.jenkinsfile | opensearch-build/jenkins/opensearch-dashboards/ | TestOpenSearchDashboardsIntegTest.groovy 4 | bwc-test.jenkinsfile | opensearch-build/jenkins/opensearch/ | TestOpenSearchBwcTest.groovy 5 | distribution-build.jenkinsfile | opensearch-build/jenkins/opensearch/ |   6 | integ-test.jenkinsfile | opensearch-build/jenkins/opensearch/ | TestOpenSearchIntegTest.groovy 7 | perf-test.jenkinsfile | opensearch-build/jenkins/opensearch/ | TestRunNonSecurityPerfTestScript.groovy 8 | maven-sign-release.jenkinsfile | opensearch-build/jenkins/opensearch-maven-release/ | TestMavenSignReleaseJob.groovy 9 | opensearch-ruby.jenkinsfile | opensearch-build/jenkins/opensearch-ruby/ |   10 | promote-artifacts.jenkinsfile | opensearch-build/jenkins/promotion/ |   11 | promote-docker-ecr.jenkinsfile | opensearch-build/jenkins/promotion/ | TestPromoteContainer.groovy 12 | promote-yum-repos.jenkinsfile | opensearch-build/jenkins/promotion/ |   13 | release-notes-check.jenkinsfile | opensearch-build/jenkins/release-notes-check/ |   14 | release-tag.jenkinsfile | opensearch-build/jenkins/release-tag/ |   15 | rpm-validation.jenkinsfile | opensearch-build/jenkins/rpm-validation/ |   16 | sign-standalone-artifacts.jenkinsfile | opensearch-build/jenkins/sign-artifacts/ | TestSignStandaloneArtifactsJob.groovy 17 | whitesource-scan.jenkinsfile | opensearch-build/jenkins/vulnerability-scan/ |   18 | perf-test.jenkinsfile | opensearch-build/jenkins/cross-cluster-replication/ | TestCCRPerfTest.groovy 19 | release-data-prepper-all-artifacts.jenkinsfile | opensearch-build/jenkins/data-prepper/ | TestDataPrepperReleaseArtifacts.groovy 20 | docker-build.jenkinsfile | opensearch-build/jenkins/docker/ |   21 | docker-copy.jenkinsfile | opensearch-build/jenkins/docker/ |   22 | gradle-check.jenkinsfile | opensearch-build/jenkins/gradle/ |   23 | check-for-build.jenkinsfile | opensearch-build/jenkins/ |  

• [ ] distribution-build.jenkinsfile at opensearch-build/jenkins/opensearch-dashboards/
• [ ] distribution-build.jenkinsfile at opensearch-build/jenkins/opensearch/
• [ ] opensearch-ruby.jenkinsfile at opensearch-build/jenkins/opensearch-ruby/
• [ ] promote-artifacts.jenkinsfile at opensearch-build/jenkins/promotion/
• [ ] promote-yum-repos.jenkinsfile at opensearch-build/jenkins/promotion/
• [x] release-notes-check.jenkinsfile at opensearch-build/jenkins/release-notes-check/
• [ ] release-tag.jenkinsfile at opensearch-build/jenkins/release-tag/
• [ ] rpm-validation.jenkinsfile at opensearch-build/jenkins/rpm-validation/
• [x] whitesource-scan.jenkinsfile at opensearch-build/jenkins/vulnerability-scan/
• [x] docker-build.jenkinsfile at opensearch-build/jenkins/docker/
• [x] docker-copy.jenkinsfile at opensearch-build/jenkins/docker/
• [ ] gradle-check.jenkinsfile at opensearch-build/jenkins/gradle/
• [ ] check-for-build.jenkinsfile at opensearch-build/jenkins/

[jordarlu]

Hi, Adding #2713 and #2714 to address test job for release-notes-check jenkinsfile. Thanks.

[jordarlu]

Hi, Adding https://github.com/opensearch-project/opensearch-build/issues/2722 and https://github.com/opensearch-project/opensearch-build/pull/2723 to address test job for whitesource-scan jenkinsfile. Thanks.

[jordarlu]

Hi, Adding #2789 and #2790 to addess test job for docker-build.jenkinsfile. thanks.

[jordarlu]

Hi, Adding https://github.com/opensearch-project/opensearch-build/issues/2791 and https://github.com/opensearch-project/opensearch-build/pull/2792 to addess test job for docker-build.jenkinsfile. thanks.