Multiple test runs creating multiple comments

[AlessioGr]

I have a test matrix, and after each matrix run it creates the report. This also means each matrix generates a different JSON file. However, for each matrix run it's creating a new comment:

which ends up creating like 15 comments.

How do I ensure this only creates one comment for all test runs?

I have tried this with a different report tag per e2e run, and with identical report tags. Doesn't change anything.

Here is my actions config:

  tests-e2e:
    runs-on: ubuntu-latest
    needs: build
    strategy:
      fail-fast: false
      matrix:
        # find test -type f -name 'e2e.spec.ts' | sort | xargs dirname | xargs -I {} basename {}
        suite:
          - _community
          - access-control
          - admin
          - auth
          - field-error-states
          - fields-relationship
          - fields
          - fields__collections__Blocks
          - fields__collections__Array
          - fields__collections__Relationship
          - fields__collections__RichText
          - fields__collections__Lexical
          - live-preview
          - localization
          - plugin-cloud-storage
          - plugin-form-builder
          - plugin-nested-docs
          - plugin-seo
          - versions
          - uploads
    env:
      SUITE_NAME: ${{ matrix.suite }}
    steps:
      # https://github.com/actions/virtual-environments/issues/1187
      - name: tune linux network
        run: sudo ethtool -K eth0 tx off rx off

      - name: Setup Node@${{ env.NODE_VERSION }}
        uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Install pnpm
        uses: pnpm/action-setup@v3
        with:
          version: ${{ env.PNPM_VERSION }}
          run_install: false

      - name: Restore build
        uses: actions/cache@v4
        timeout-minutes: 10
        with:
          path: ./*
          key: ${{ github.sha }}-${{ github.run_number }}

      - name: Start LocalStack
        run: pnpm docker:start
        if: ${{ matrix.suite == 'plugin-cloud-storage' }}

      - name: Store Playwright's Version
        run: |
          # Extract the version number using a more targeted regex pattern with awk
          PLAYWRIGHT_VERSION=$(pnpm ls @playwright/test --depth=0 | awk '/@playwright\/test/ {print $2}')
          echo "Playwright's Version: $PLAYWRIGHT_VERSION"
          echo "PLAYWRIGHT_VERSION=$PLAYWRIGHT_VERSION" >> $GITHUB_ENV

      - name: Cache Playwright Browsers for Playwright's Version
        id: cache-playwright-browsers
        uses: actions/cache@v3
        with:
          path: ~/.cache/ms-playwright
          key: playwright-browsers-${{ env.PLAYWRIGHT_VERSION }}

      - name: Setup Playwright - Browsers and Dependencies
        if: steps.cache-playwright-browsers.outputs.cache-hit != 'true'
        run: pnpm exec playwright install --with-deps chromium

      - name: Setup Playwright - Dependencies-only
        if: steps.cache-playwright-browsers.outputs.cache-hit == 'true'
        run: pnpm exec playwright install-deps chromium

      - name: E2E Tests
        run: PLAYWRIGHT_JSON_OUTPUT_NAME=results_${{ matrix.suite }}.json pnpm test:e2e ${{ matrix.suite }}
        env:
          PLAYWRIGHT_JSON_OUTPUT_NAME: results_${{ matrix.suite }}.json
          NEXT_TELEMETRY_DISABLED: 1

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results-${{ matrix.suite }}
          path: test/test-results/
          if-no-files-found: ignore
          retention-days: 1

      - uses: daun/playwright-report-summary@v3
        with:
          report-file: results_${{ matrix.suite }}.json
          report-tag: ${{ matrix.suite }}
          job-summary: true

[daun]

Hi :) The action requires a single json report output by Playwright. If you're running multiple matrixes or shards, the official strategy is to create the reports as separate blob reports, then combine them using npx playwright merge-reports. You can find an example of how this can be done in the swup project's end-to-end test workflow. Also see the Playwright docs on merging reports.

[daun]

@AlessioGr I'll close this as solved, feel free to comment if the above solution doesn't work for you.