Playwright | Generic Testomatio reporter error prevents screenshots from being added to Testomatio test runs

[mascarion]

Describe the bug After test execution a generic error message appears related to the reporter. The S3 bucket has been configured correctly and was previously populated with screenshots successfully. Can't think of any changes made recently that would cause the issue. The main change is that allure report is now also used in addition to the default html reporter and Testomatio reporter.

To Reproduce Execute Playwright tests via Gitlab pipeline (assume other pipelines will have same behaviour). The default Playwright html reporter generates artifacts as well as Testomatio and Allure. Both the other reporters include screenshots.

Extract from YAML file:

Extract from playwright.config.ts:

Expected behaviour No reporter error messages.

Actual behaviour

Desktop

• OS: Ubuntu 22.04.3 LTS
• Browser: N/A
• Pipeline: Gitlab
• Automation tool: Playwright
• Testomatio version: 1.0.17

[olexandr13]

@mascarion

1. remove exclamation mark from env var

2. this is allure reporter error, not Testomatio reporter

[mascarion]

@mascarion

1. remove exclamation mark from env var
2. this is allure reporter error, not Testomatio reporter

The exclamation mark is unrelated to this issue and in Typescript the ! operator tells the compiler to ignore the possibility of it being undefined. I removed it anyway and no difference in behaviour. The output states that 51 files were uploaded however looking in the S3 bucket nothing has been uploaded.

[olexandr13]

@mascarion could you please attach the screenshot from Testomatio where the absence of artifacts could be seen?

[mascarion]

@olexandr13 I don't know if it is relevant but the latest test run in Testomat shows as still running even though it completed over 30 mins ago.

Below is a screenshot of a successful artifact upload. The thing is this was with just a few tests, running all tests causes the issue.

You will also notice that the number of completed tests runs with a large number of tests plummeted around the 8th of Feb. This is not because the test runs did not complete, it is just case of Testomat still believing they are running even though they already completed. Only test runs with a handful of tests completed according to Testomat.

Looking at any completed test in the latest run no artifact is present:

[mascarion]

@mascarion

1. remove exclamation mark from env var
2. this is allure reporter error, not Testomatio reporter

Please see below for evidence that the error is related to Testomat. I reran a different pipeline this morning (the last one that successfully uploaded to our S3 bucket). So when the upload is successful the highlighted entries would exist in the logs. That is missing from the original logs I provided a screenshot of

To note is that the pipeline that passed had only 3 tests in it. The pipeline that is not completing (according to Testomatio) has 51 tests so the number of S3 uploads attempted may be related. The pipeline does complete successfully in Gitlab after 20 mins however Testomatio app shows it as still running until eventually timing out.

Error reported originally:

[tikolakin]

Hi @mascarion , do you run tests in parallel? If so, I would suggest you don't rely on S3 credentials provided by testomatio as there seems to be an issue indeed. Instead, if you try to make S3 credentials available in execution env - testomatio should be able to upload them. Please let me know how it went if that is your usecase Thanks

[mascarion]

Hi @tikolakin actually I have set tests specifically to only use one worker as parallel execution causes issues with the SUT.

This is what is present in the YAML file which is used for pipeline runs: S3_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID} S3_BUCKET=${S3_BUCKET} S3_REGION=${S3_REGION} S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY} MANAGER_PASSWORD=${MANAGER_PASSWORD} MANAGER_EMAIL_ADDRESS=${MANAGER_EMAIL_ADDRESS} URL=${URL} TESTOMATIO=${TESTOMATIO} TESTOMATIO_PRIVATE_ARTIFACTS=${TESTOMATIO_PRIVATE_ARTIFACTS} npx playwright test --workers=1 && npx allure generate --single-file ./allure-results

My previous comments mention that the S3 uploads work fine with a smaller subset of tests. If only 3 tests are executed upload works fine. If I run 51 tests however no upload takes place. Since neither is executed in parallel they should both perform upload successfully.

[olexandr13]

@mascarion could you please attach the CI job pipeline log? Is this problem reproduced locally?

Also I would ask you to add this env var DEBUG=@testomatio/reporter:file-uploader, run tests and show us the logs.

[olexandr13]

@mascarion also please install the latest reporter version (1.2.0+). We had similar issues previously and I hope its already fixed.

[mascarion]

@olexandr13 I just updated the reporter to v1.2.0 and the s3 upload was successful. Thanks for all your help.