search

DataDog / synthetics-ci-github-action

Use Browser and API tests in your CI/CD with Datadog Continuous Testing
https://docs.datadoghq.com/continuous_testing/
Apache License 2.0
41 stars 14 forks source link

Synthetic tests in the datadog passes but the workflow using the datadog github action fails #165

Closed shalini171993 closed 1 year ago

shalini171993 commented 1 year ago

Hi ,

I have been using DataDog/synthetics-ci-github-action@v0.14.0 for triggering the datadog tests from github action workflow. The datadog test passes but the workflow fails. I saw there has been a similar issue reported already - https://github.com/DataDog/synthetics-ci-github-action/issues/103. The workflow fails with "failed: 1" and not because of timeout. However in the datadog it shows status as timed out. Is there any fix for this ?

Drarig29 commented 1 year ago

Hey @shalini171993! Thanks for reporting an issue!

Can you provide the logs of DataDog/synthetics-ci-github-action@v0.14.0? Don't forget to redact any information you don't want to share.

shalini171993 commented 1 year ago

Hey @Drarig29 Thanks for your reply. Sorry that I cant provide the logs. But the error looks like below :

=== REPORT === ✖ [blocking] [test] - location: --------- - device: ----------- ⎋ Total duration: 39924 ms - View test run details: https://xxxxxxxxxxxx/synthetics/details/xxxxxxxxxxx/result/xxxxxxxxxx?from_ci=true (not yet received) ⇢ | 10 skipped steps View full summary in Datadog: https://xxxxxxxxxxxxxxxx/synthetics/explorer/ci?batchResultId=xxxxxxxxxxxxxxxxxxxxxxx Continuous Testing Summary: Test Results: 0 passed, 1 failed

[error]Datadog Synthetics tests failed: criticalErrors: 0, passed: 0, failedNonBlocking: 0, failed: 1, skipped: 0, notFound: 0, timedOut: 0

But the datadog test has passed.All the steps have been executed but in the log it shows the steps have been skipped

Drarig29 commented 1 year ago

Thank you! Do you have retries configured in Datadog for this test? (cf. docs)

If you do, then you will probably see that the test has fast retried, so it eventually succeeded.

When you follow the link for View full summary in Datadog, do you see any text saying something like this?

PASSED 2 test runs (1 test run + 1 fast retry)"

shalini171993 commented 1 year ago

I hadnt setup retry before. But now I tried setting the retry count as 2 for the datadog test.

=== REPORT ===

✓ [xxx-xxx-xxx] [test] - location: --------- - device: ----------- ⎋ Total duration: 37594 ms - View test run details: https://xxxxxxxxxx/synthetics/details/xxxxxxxxx/result/xxxxxxxxxxxxxxxx?from_ci=true (not yet received) ⇢ | 10 skipped steps

View full summary in Datadog: https://xxxxxxxxxxx/synthetics/explorer/ci?batchResultId=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Continuous Testing Summary: Test Results: 1 passed, 0 failed (1 timed out) Max parallelization configured: XXXX tests running at the same time Total Duration: 10m 7s

Increase your parallelization to reduce your total duration: https://Xxxxxxxxx/synthetics/settings/continuous-testing Because failOnTimeout is disabled, the command will succeed. Use failOnTimeout: true to make it fail instead. Datadog Synthetics tests succeeded: criticalErrors: 0, passed: 1, failedNonBlocking: 0, failed: 0, skipped: 0, notFound: 0, timedOut: 1 Results URL: https://xxxxxxxxxxx/synthetics/explorer/ci?batchResultId=xxxxxxxxxxxxxxx

The datadog test has passed as usual. Clicking on the Results URL shows the test has failed with timeout FAILED1 test run (1 test run + 0 fast retries) Looks like there has been no retry.But strange that it shows passed: 1,timedOut: 1 in the log.

I tried to run the workflow couple of times and the results are the same.

Drarig29 commented 1 year ago

The Total Duration: 10m 7s makes it sound like you have set up a 10 minute timeout. Have you set a polling_timeout for the github action?

With the release of Continuous Testing, tests running in CI now run sequentially by default. In order to make your pipeline faster, you can change your parallelization setting in the Continuous Testing Settings page, as mentioned in the output.

shalini171993 commented 1 year ago

Yes I have set the polling timeout . But its strange that the test gets timed out every time. Are you able to reproduce this issue ? Do you have any work around for this please?

Drarig29 commented 1 year ago

Hey @shalini171993!

For your last issue, namely:

But its strange that the test gets timed out every time. Are you able to reproduce this issue?

Thanks for reporting it, it should be fixed now. This was not related to the polling timeout.

If you still have problems, please create a support ticket here. Closing this issue.