Update CI workflows for E2E tests

[lukaszcl]

This PR introduces a Reusable GitHub Workflow for E2E tests, centralizing all the necessary logic to run various types of tests—such as Docker and K8s remote runner—within a single workflow. This approach allows other E2E test workflows to utilize this central workflow at different stages of the software development process, whether it's for every commit, within the merge queue, nightly, or on-demand.

The workflow sources the tests to run from the .github/e2e-tests.yml file, where each test or group of tests is defined with specific parameters: test-cmd to dictate how to execute the test, runs-on to specify the required GitHub runner, and a workflows parameter that outlines when each test should be executed (e.g., on every pull request or nightly).

  # Example of 1 runner for all tests in integration-tests/smoke/ocr_test.go
  - id: smoke/ocr_test.go:* 
    path: integration-tests/smoke/ocr_test.go
    test-type: docker
    test-inputs:
       chainlinkImage: '{{ env.QA_CHAINLINK_IMAGE }}'
       .. # other test inputs if required by a test
    runs-on: ubuntu-latest
    workflows:
      - Run PR E2E Tests
      - Run Nightly E2E Tests
    test-cmd: cd integration-tests/ && go test smoke/ocr_test.go -timeout 30m -count=1 -json
    pyroscope-env: ci-smoke-ocr-evm-simulated-nightly

The run-e2e-tests-reusable-workflow.yml workflow can be utilized in various ways:

• Specify Test IDs: Provide comma-separated test IDs in the test_ids input to run specific tests. For example, see the run-selected-e2e-tests.yml workflow.
• Run Tests by Workflow: Use the test_workflow input to run all tests associated with a particular workflow. For example, see the run-nightly-e2e-tests.yml workflow.
• Custom Test List: Input a list of tests with custom settings using the test_list input. For example, see the run-automation-ondemand-e2e-tests.yml workflow.

To learn more about new CI Workflows for E2E Tests, please visit https://smartcontract-it.atlassian.net/wiki/spaces/TT/pages/815497220/CI+Workflows+for+E2E+Tests

Note: This update does not affect existing workflows for now. Changes to integrate existing workflows with this new setup will be addressed in separate PRs. More details are provided below.

---

Full scope of this PR:

1. Add reusable workflow to run E2E tests: run-e2e-tests-reusable-workflow.yml
2. Add a workflow to run any E2E test from .github/e2e-tests.yml on demand: run-selected-e2e-tests.yml

   gh workflow run run-selected-e2e-tests.yml --ref ${branch_name} -f chainlink_version=develop -f test_ids="smoke/ocr_test.go:*,soak/ocr_test.go:^TestOCRv1Soak$"

3. Add a workflow to run tests nightly: run-nightly-e2e-tests.yml. Tests to run nightly have "Run Nightly E2E Tests" in workflows field in .github/e2e-tests.yml file. Example build.
4. Add run-automation-ondemand-e2e-tests.yml that works the same as automation-ondemand-tests.yml but uses the reusable workflow. Once it's tested we can remove the old workflow. run-automation-ondemand-e2e-tests.yml is an example how to run tests with custom test inputs
5. Add go tool to detect dead-tests (tests that are not part of any CI workflow). Learn more here. This is disabled by default for now
6. Add go tool to create default test config override (./.github/actions/create-default-e2e-config-override) and use it instead of the old bash script
7. Add jobs to reusable workflow to easily build required Chainlink Node image and Chainlink Plugins image. Required versions can be set using require_chainlink_image_versions_in_qa_ecr and require_chainlink_plugin_versions_in_qa_ecr workflow inputs

Not in the scope:

1. Refactoring existing workflows to utilise the new Reusable E2E Tests Workflow will be addressed in separate PRs
2. Allow to pass full test_config_base64_override as workflow input. This will be enabled once TestConfig does not have any secrets. Related ticket https://smartcontract-it.atlassian.net/browse/TT-1283
3. Add Slack notifications for workflows that use the E2E tests reusable workflow. This will be addressed in

[github-actions[bot]]

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[cl-sonarqube-production[bot]]

Quality Gate passed

Issues
23 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube