Artifacts are downloaded on v0.183.0 only if at least a single failure occurred during the run

SmialyKot commented 3 weeks ago

Description

After updating to the latest saucectl in saucectl-run-action downloading artifacts became flaky - sometimes it downloads the artifacts as specified in config, usually it doesn't. We upload those artifacts to gh actions after finishing the test.

Config to reproduce

This is how we declare artifact download in config.yml:

artifacts:
  download:
    when: always
    match:
      - junit.xml
    directory: ./test-results/saucelabs-results/
  cleanup: true

And this is how we run our Espresso tests on real Android devices via Github Actions:

- name: Run tests
  uses: saucelabs/saucectl-run-action@v4
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  with:
    timeout: 90m
    saucectl-version: v0.183.0

- name: Upload tests result
  if: ${{ (success() || failure()) }}
  uses: actions/upload-artifact@v4
  with:
    name: some_unique_name
    path: test-results/
    retention-days: 2
    overwrite: true

The following warning is issued by the upload-artifact step:

No files were found with the provided path: test-results/. No artifacts will be uploaded.

The interesting part is that sometimes results are downloaded - we run 6 jobs in parallel, some of them end up uploading results, most of them don't. They are not always the same jobs.

Desired behavior

After reverting saucectl to v0.182.0 everything goes back to normal - all test results are always downloaded and uploaded as expected. This is a regression strictly in v0.183.0 version.

Versions

saucectl-run-action@v4
saucectl v0.183.0
Ubuntu 20.04

SmialyKot commented 3 weeks ago

After further investigation it turns out that download is not flaky, it's just that the junit.xml result is only downloaded if there was at least a single failure during testing - we use smartRetry for failed tests so the job itself was successful. Since we specify to always download test results it makes no sense that they are downloaded only if there was a failure during the run.

alexplischke commented 1 week ago

Addressed via https://github.com/saucelabs/saucectl/pull/943. Thank you for reporting this issue!

saucelabs / saucectl