hydephp / develop

The HydePHP Source Code Monorepo
https://hydephp.com
MIT License
13 stars 6 forks source link

Internal: Parallel smoke testing #1928

Closed caendesilva closed 2 months ago

caendesilva commented 2 months ago

We've updated our smoke tests workflow to run tests in parallel, improving efficiency while maintaining compatibility with our existing processes. Here are the key changes:

  1. Renamed the workflow to "🔥 Parallel Smoke Tests" to reflect its new parallel nature.
  2. Added a step to prepare separate directories for unit and feature tests.
  3. Introduced a "Run Parallel Smoke Tests" step that:
    • Defines a function run_tests to execute tests for each suite.
    • Runs unit and feature tests in parallel using background processes.
    • Uses sed to prefix each line of output with either [UNIT] or [FEATURE] for clarity, with proper padding for alignment.
    • Generates separate JUnit XML reports for each test suite.
  4. Added a "Merge JUnit XML Reports" step that combines the unit and feature test reports into a single report.xml file, ensuring compatibility with existing ping scripts.
  5. Retained the rest of the workflow, using the combined report.xml for subsequent steps.
  6. Removed the separate CI smoke test feature as this parallel approach is now fast enough.

This parallel approach significantly speeds up the smoke tests while maintaining compatibility with our existing CI processes. We get interleaved, real-time output from both test suites.

graph TD
  A[Start] --> B[Set up environment]
  B --> C[Checkout code]
  C --> D[Validate composer files]
  D --> E[Cache Composer packages]
  E --> F[Install dependencies]
  F --> G[Prepare test directories]
  G --> H[Run Unit Tests]
  G --> I[Run Feature Tests]
  H --> J[Generate Unit Test Report]
  I --> K[Generate Feature Test Report]
  J --> L[Merge JUnit XML Reports]
  K --> L
  L --> N[Ping statistics server]
  N --> O[End]
  style H fill:#f9f,stroke:#333,stroke-width:2px,color:#333
  style I fill:#f9f,stroke:#333,stroke-width:2px,color:#333
  style J fill:#ccf,stroke:#333,stroke-width:2px,color:#333
  style K fill:#ccf,stroke:#333,stroke-width:2px,color:#333
  style L fill:#fcc,stroke:#333,stroke-width:2px,color:#333
caendesilva commented 2 months ago

Baseline test execution time https://github.com/hydephp/develop/pull/1928/commits/2337a942155d007381a612fe32455b00a5d4e0bf

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (305a6fc) to head (766b7e6). Report is 4 commits behind head on master.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #1928 +/- ## =========================================== Coverage 100.00% 100.00% Complexity 1766 1766 =========================================== Files 182 182 Lines 4707 4707 =========================================== Hits 4707 4707 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

caendesilva commented 2 months ago

Merging coverage adds a lot of complexity, so pivoting to do smoke tests in parallel as that's where the biggest pain point is anyway, and we don't need to assemble so much data.

Baseline execution time (spot test) 26s for test execution time 34s for test succeeded time

New execution time (spot test) 20s for test execution time 31s for test succeeded time