EISDIR: illegal operation on a directory, read

[poponuts]

I get this error:

node:fs:744
  handleErrorFromBinding(ctx);
  ^

Error: EISDIR: illegal operation on a directory, read
    at Object.readSync (node:fs:744:3)
    at tryReadSync (node:fs:444:20)
    at Object.readFileSync (node:fs:490:19)
    at ResultsParser.parseFromJsonFile (/home/runner/.npm/_npx/912153[8](https://github.com/.../actions/runs/7176906992/job/19542812131?pr=7957#step:8:9)dc03c[9](https://github.com/.../actions/runs/7176906992/job/19542812131?pr=7957#step:8:10)62b/node_modules/playwright-slack-report/dist/src/ResultsParser.js:39:25)
    at Command.<anonymous> (/home/runner/.npm/_npx/9[12](https://github.com/.../actions/runs/7176906992/job/19542812131?pr=7957#step:8:13)[15](https://github.com/.../actions/runs/7176906992/job/19542812131?pr=7957#step:8:16)38dc03c962b/node_modules/playwright-slack-report/dist/cli.js:32:47) {
  errno: -21,
  syscall: 'read',
  code: 'EISDIR'
}

by running the following command: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j blob-report/

The artifact is created based on this in my playwright.yml file:

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: all-blob-reports
        path: blob-report

    - name: Merge into JSON Report
      run: npx playwright merge-reports --reporter json blob-report/

    - name: Upload JSON Report
      uses: actions/upload-artifact@v3
      with:
        name: json-report--attempt-${{ github.run_attempt }}
        path: blob-report
        retention-days: 3

---

When I tried including the actual json file, it says the artifact can't be found: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j blob-report/json-report--attempt-${{ github.run_attempt }}.json

Exact error: ❌ JSON results file does not exist: blob-report/json-report--attempt-1/: Use --json-results e.g. --json-results="./results.json"

[ryanrosello-og]

thanks for raising, let me try and repro

[ryanrosello-og]

does it complain if you feed in the full absolute path? Not sure how you can derive that in GH actions

[poponuts]

On this step:

    - name: Upload JSON Report
      uses: actions/upload-artifact@v3
      with:
        name: json-report--attempt-${{ github.run_attempt }}
        path: blob-report
        retention-days: 3

I get something like this:

Run actions/upload-artifact@v3
With the provided path, there will be 12 files uploaded
Starting artifact upload
For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging
Artifact name is valid!
Container for artifact "json-report--attempt-1" successfully created. Starting upload of file(s)
Uploaded /home/runner/work/.../blob-report/report-1.zip (16.6%) bytes 0:8388607
Uploaded /home/runner/work/.../blob-report/resources/60d567cf8bb2a0[9] (91.2%) bytes 0:8388607
...
Total size of all the files uploaded is 100954095 bytes
File upload process has finished. Finalizing the artifact upload
Artifact has been finalized. All files have been successfully uploaded!

The raw size of all the files that were specified for upload is 101550406 bytes
The size of all the files that were uploaded is 100954095 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage

Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 

Artifact json-report--attempt-1 has been successfully uploaded!

---

... then I get that error on the next step:

    - name: Send to Slack
      run: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j blob-report/

[poponuts]

does it complain if you feed in the full absolute path? Not sure how you can derive that in GH actions

let me try this.

[ryanrosello-og]

@poponuts also try viewing the contents of the artifact... just to make the json file exists

steps:
- uses: actions/checkout@v3

- uses: actions/download-artifact@v3
  with:
    path: path/to/artifacts

- name: Display structure of downloaded files
  run: ls -R
  working-directory: path/to/artifacts

[poponuts]

Run ls -R
  ls -R
  shell: /usr/bin/bash -e {0}
  env:
    NODE_OPTIONS: --max-http-header-size=[4](https://github.com/xxx/actions/runs/7188206823/job/19577691114?pr=7957#step:6:4)09[6](https://github.com/xxx/actions/runs/7188206823/job/19577691114?pr=7957#step:6:7)
.:
report-1.zip

The 4 steps (Download, Display, Merge, Upload) works , only the Sending to Slack with the playwright-slack-report command that doesn't work:

[ryanrosello-og]

would you be able to paste in the command line arguments your are sending to the app? seems to have been cutoff in the screenshot above

[ryanrosello-og]

also I noticed the report-1.zip is downloaded. The cli app expects .json as input so you may need a step to unzip

[poponuts]

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: all-blob-reports
        path: blob-report

    - name: Display structure of downloaded files
      run: ls -R
      working-directory: blob-report

    - name: Merge into JSON Report
      run: npx playwright merge-reports --reporter json blob-report/

    - name: Upload JSON Report
      uses: actions/upload-artifact@v3
      with:
        name: json-report--attempt-${{ github.run_attempt }}
        path: blob-report
        retention-days: 3

   - name: Send to Slack
      run: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j blob-report/

not too sure with the unzip as these are standard commands in the Playwright & Github site. Also, as you can see in screenshot, seems the file has been uploaded after merge. It's probably a matter of just picking up that report.

[poponuts]

Ok, I'm progressing a bit. Seems like an absolute path is needed and can't pick up on relative path with this command: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j /home/runner/work/blob-report/report-${{ github.run_attempt }}.jsonl

As per https://playwright.dev/docs/test-sharding#github-actions-example and my commands above, it seems merging reports would produce a file called report-1.jsonl only. So running the above, it produced the error below:

undefined:2
{"method":"onConfigure","params":{"config":{"configFile":"playwright.config.ts","globalTimeout":0,"maxFailures":0,"metadata":{"actualWorkers":1},"rootDir":"/home/runner/work","version":"1.40.0","workers":1,"listOnly":false}}}
^

SyntaxError: Unexpected token { in JSON at position 15[7](https://github.com/xxx/actions/runs/7189002799/job/19580092787?pr=7957#step:10:8)
    at JSON.parse (<anonymous>)
    at ResultsParser.parseFromJsonFile (/home/runner/.npm/_npx/912153[8](https://github.com/xxx/actions/runs/7189002799/job/19580092787?pr=7957#step:10:9)dc03c[9](https://github.com/xxx/actions/runs/7189002799/job/19580092787?pr=7957#step:10:10)62b/node_modules/playwright-slack-report/dist/src/ResultsParser.js:40:33)
    at Command.<anonymous> (/home/runner/.npm/_npx/9[12](https://github.com/xxx/actions/runs/7189002799/job/19580092787?pr=7957#step:10:13)1538dc03c962b/node_modules/playwright-slack-report/dist/cli.js:32:47)

[ryanrosello-og]

would you be able to print out the contents of the .jsonl file?

[poponuts]

too big to share here but I found this (so it could be a matter of formatting): JSONL is a text-based format that uses the . jsonl file extension and is essentially the same as JSON format except that newline characters are used to delimit JSON data. It also goes by the name JSON Lines.

Also not sure why its producing this file using the merge report command in Playwright when the reporting is supposed to be json: npx playwright merge-reports --reporter json blob-report/

[ryanrosello-og]

hmm, this makes me sad. Leave it with me

[poponuts]

After upload, I tried to download it to the folder too with the following:

    - name: Download merged report from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: json-report--attempt-${{ github.run_attempt }}
        path: blob-report

It was successfully downloaded: Artifact json-report--attempt-1 was downloaded to /home/runner/work/blob-report

However, with the step: SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j /home/runner/work/blob-report/json-report--attempt-1

It errored with: JSON results file does not exist: /home/runner/work/blob-report/json-report--attempt-1

[ryanrosello-og]

yeah it expects a json file, not a folder. wonder if you could coerce it into generating a json file (Not a .jsonl)

in the merge step, try:

npx playwright merge-reports --reporter json blob-report/send_this_to_slack.json

and invoke the cli using something like:

SLACK_BOT_USER_OAUTH_TOKEN=xxx npx playwright-slack-report -c playwright/cli_config.json -j /home/runner/work/blob-report/send_this_to_slack.json

weird how its not complaining about the config file 🤔 but I guess that would be the next problem once it gets a hold of a valid results json file

[poponuts]

I think the merge-reports command expects a folder instead as from the above, I got the error:

Run npx playwright merge-reports --reporter json blob-report/send_this_to_slack.json
npm WARN exec The following package was not found and will be installed: playwright@1.40.1
Error: Directory does not exist: /home/runner/work/blob-report/send_this_to_slack.json
    at mergeReports (/home/runner/.npm/_npx/e41f203b[7](https://github.com/xxx/actions/runs/7190367703/job/19583582528?pr=7957#step:7:8)505f1fb/node_modules/playwright/lib/cli.js:173:23)
    at async t.<anonymous> (/home/runner/.npm/_npx/e41f203b7505f1fb/node_modules/playwright/lib/cli.js:[9](https://github.com/xxx/actions/runs/7190367703/job/19583582528?pr=7957#step:7:10)3:7)
Error: Process completed with exit code 1.

I also tried to run it locally with the json file generated from the Upload JSON Report command and it errored (but this is a separate issue):

node_modules/playwright-slack-report/dist/src/ResultsParser.js:41
        const { retries } = parsedData.config.projects[0];
                ^

TypeError: Cannot destructure property 'retries' of 'parsedData.config.projects[0]' as it is undefined.
    at ResultsParser.parseFromJsonFile (/xxx/node_modules/playwright-slack-report/dist/src/ResultsParser.js:41:17)
    at Command.<anonymous> (/xxx/node_modules/playwright-slack-report/dist/cli.js:32:47)

It seems there is error reading 'retries' but I didnt' define projects as part of my playwright.config.ts file.

[ryanrosello-og]

does the downloaded json file structure/data resemble something this ? https://github.com/ryanrosello-og/playwright-slack-report/blob/main/tests/test_data/valid_test_results.json

[ryanrosello-og]

@poponuts try updating to the latest.

Couple things:

• TypeError: Cannot destructure property 'retries' of 'parsedData.config.projects[0]' as it is undefined. was legit issue, the app wasn't handling situations where retry is not configured in the Playwright config
• Redirection is required when merging results 🤦

❌ This won't work: npx playwright merge-reports --reporter json ./all-blob-reports

✅ This should work: npx playwright merge-reports --reporter json ./all-blob-reports > merged_tests_results.json

here is a full sample of my working GH action

name: Playwright Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  playwright-tests:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1/2, 2/2]
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 18
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright browsers
      run: npx playwright install --with-deps

    - name: Run Playwright tests
      run: npx playwright test --shard ${{ matrix.shard }}

    - name: Upload blob report to GitHub Actions Artifacts
      if: always()
      uses: actions/upload-artifact@v3
      with:
        name: all-blob-reports
        path: blob-report
        retention-days: 1

  merge-reports:
    # Merge reports after playwright-tests, even if some shards have failed
    if: always()
    needs: [playwright-tests]

    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-node@v3
      with:
        node-version: 18
    - name: Install dependencies
      run: npm ci

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v3
      with:
        name: all-blob-reports
        path: all-blob-reports

    - name: Merge into JSON Report
      run: npx playwright merge-reports --reporter json ./all-blob-reports > merged_tests_results.json

    - name: List files
      run: ls -la

    - name: The path
      run: echo ${GITHUB_WORKSPACE}

    - name: View merged results
      run: cat ${GITHUB_WORKSPACE}/merged_tests_results.json

    - name: Send report to Slack using CLI
      env:
        SLACK_BOT_USER_OAUTH_TOKEN: ${{ secrets.SLACK_BOT_USER_OAUTH_TOKEN }}
      run: npx playwright-slack-report --config="${GITHUB_WORKSPACE}/cli_config.json" --json-results="${GITHUB_WORKSPACE}/merged_tests_results.json"

[poponuts]

Firstly, thanks @ryanrosello-og for spending this much time on this issue. Appreciate it. Anyway, I updated to latest and followed your steps but I get this error on the merge command: Download blob reports from Github Actions Artifacts:

Starting download for all-blob-reports
Directory structure has been setup for the artifact
Total number of files that will be downloaded: 14
Artifact all-blob-reports was downloaded to /home/runner/work/all-blob-reports
Artifact download has finished successfully

List files (I included this step to check if the report was downloaded):

Run ls -la
drwxr-xr-x  9 runner docker 4096 Dec 14 00:48 all-blob-reports

Merge into JSON Report:

Run npx playwright merge-reports --reporter json ./all-blob-reports > merged_tests_results.json

npm WARN exec The following package was not found and will be installed: playwright@1.40.1
Error: Directory does not exist: /home/runner/work/all-blob-reports
    at mergeReports (/home/runner/.npm/_npx/e41f203b[7](https://github.com/xxx/actions/runs/7202806638/job/19622026466?pr=7957#step:7:8)505f1fb/node_modules/playwright/lib/cli.js:173:23)
    at async t.<anonymous> (/home/runner/.npm/_npx/e41f203b7505f1fb/node_modules/playwright/lib/cli.js:[9](https://github.com/xxx/actions/runs/7202806638/job/19622026466?pr=7957#step:7:10)3:7)
Error: Process completed with exit code 1.

Also, I have retries on my playwright.config.ts but I don't have it under projects:

 // Retry on CI only.
 retries: process.env.CI ? 1 : 0,

[poponuts]

Hi @ryanrosello-og I got it working. Just changed some directories. Thanks for all the help, the config worked too without changing anything.