`junit upload` - Test suites in certain test results directories are not shown on CI Test UI despite successful upload #923

Closed ImFromMarzKa closed 1 year ago

Summary

I am running datadog-ci junit upload on multiple test report directories using a single command. The log output from the command shows that all of the expected JUnit xml files were uploaded successfully.

Expected Behavior: All test results are shown in DataDog CI Test UI after successful upload.

Actual Behavior: Test results from 23 out of 27 test suites are not shown in the DataDog CI Test UI, even though datadog-ci logs show successful upload of these test results.

The reports for all of the missing test suites share the same directory structure (**/build/test-results/test). All missing test suites are executed by the same Gradle task (./gradlew test). The other test suites that are shown in the UI are all executed by different Gradle tasks and their reports are in different directories.

Details

CLI Version

datadog-ci version
v2.14.0

CLI Installation

Installed via yarn.

Initially, datadog-ci was installed on the Docker image used by the GitLab job like this:

FROM {domain}.jfrog.io/node:14-buster-slim

RUN yarn global add @datadog/datadog-ci
RUN apt update && apt install wget unzip -y

RUN yarn add --dev @datadog/datadog-ci

Once I ran into issues, I tried installing directly from the GitLab job script so that I had more control, like this:

datadog:publish-junit-multiple:
  image: {domain}.jfrog.io/node:14-buster-slim
  {...other job config...}
  script:
    - yarn global add @datadog/datadog-ci
    - apt update && apt install wget unzip -y
    - yarn add --dev @datadog/datadog-ci
    - datadog-ci version

CLI Command

datadog-ci junit upload --service ${DATADOG_SERVICE} api/build/test-results/test client/build/test-results/consumerCDCTest client/build/test-results/test server/build/test-results/integrationTest server/build/test-results/providerCDCTest server/build/test-results/test

Output from CLI Command

Starting upload with concurrency 20. 
Will look for jUnit XML files in api/build/test-results/test, client/build/test-results/consumerCDCTest, client/build/test-results/test, server/build/test-results/integrationTest, server/build/test-results/providerCDCTest, server/build/test-results/test
service: {datadog_service_name}
Uploading jUnit XML test report file in api/build/test-results/test/TEST-com.company.component.api.model.IDTest.xml
Uploading jUnit XML test report file in client/build/test-results/consumerCDCTest/TEST-com.company.component.client.CredentialManagementConsumerCDCT.xml
Uploading jUnit XML test report file in client/build/test-results/consumerCDCTest/TEST-com.company.component.client.OrganizationConsumerCDCT.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.client.api.credential.CredentialManagementApiTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.client.api.organization.OrganizationApiTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.client.config.CharonServiceConfigurerTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.config.service.ServiceConfigurerTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.helper.UriBuilderTest$HostWithPort.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.helper.UriBuilderTest$TestDifferentBaseURIs.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.interceptor.Auth0ClientTokenInterceptorTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.interceptor.BasicAuthInterceptorTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.interceptor.RequestTelemetryInterceptorTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.service.HttpBuilderServiceTest.xml
Uploading jUnit XML test report file in client/build/test-results/test/TEST-com.company.component.shared.service.HttpServiceTest.xml
Uploading jUnit XML test report file in server/build/test-results/integrationTest/TEST-com.company.component.server.OrganizationControllerIT.xml
Uploading jUnit XML test report file in server/build/test-results/providerCDCTest/TEST-com.company.component.server.CharonServerProviderCDCT.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testDeactivateCredential.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGenerateCredential.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredential.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialClientSecret.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialsHistory.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.OrganizationControllerWebMvcTest.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.OrganizationControllerWebMvcTest$testCreateOrganization.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.controller.OrganizationControllerWebMvcTest$testGetOrganization.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.datastore.credential.CredentialEntityTest.xml
Uploading jUnit XML test report file in server/build/test-results/test/TEST-com.company.component.server.service.CredentialManagementServiceTest.xml
✅ Uploaded 27 files in 1.25 seconds.
Not syncing git metadata (not a git repo)
=================================================================================================
* View detailed reports on Datadog (they can take a few minutes to become available)
* Commit report:
* https://app.datadoghq.com/ci/redirect/tests/{remaining path}?env=ci
* Test runs report:
* https://app.datadoghq.com/ci/test-runs?query=%20%40ci.job.url%3A%22{remaining path}jobs%2F796073%22
=================================================================================================

Test Suites Shown in UI

com.company.component.server.CharonServerProviderCDCT
com.company.component.client.CredentialManagementConsumerCDCT
com.company.component.client.OrganizationConsumerCDCT
com.company.component.server.OrganizationControllerIT

Total Test Suites Shown: 4

Total Test Cases Shown: 21

Sample XML Report for Test Suite Shown in UI

Filename:

TEST-com.company.component.client.CredentialManagementConsumerCDCT.xml

Content:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="com.company.component.client.CredentialManagementConsumerCDCT" tests="1" skipped="0" failures="0" errors="0" timestamp="2023-06-08T17:47:22" hostname="runner-xyz" time="0.899">
  <properties/>
  <testcase name="testGenerateCredential(MockServer)" classname="com.company.component.client.CredentialManagementConsumerCDCT" time="0.899"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[[Test worker] INFO au.com.dius.pact.consumer.junit5.PactConsumerTestExt - Writing pacts out to default directory
]]></system-err>
</testsuite>

Test Suites Not Shown in UI

All test suites in directories matching: **/build/test-results/test

com.company.component.api.model.IDTest
com.company.component.client.api.credential.CredentialManagementApiTest
com.company.component.client.api.organization.OrganizationApiTest
com.company.component.client.config.CharonServiceConfigurerTest
com.company.component.shared.config.service.ServiceConfigurerTest
com.company.component.shared.helper.UriBuilderTest$HostWithPort
com.company.component.shared.helper.UriBuilderTest$TestDifferentBaseURIs
com.company.component.shared.interceptor.Auth0ClientTokenInterceptorTest
com.company.component.shared.interceptor.BasicAuthInterceptorTest
com.company.component.shared.interceptor.RequestTelemetryInterceptorTest
com.company.component.shared.service.HttpBuilderServiceTest
com.company.component.shared.service.HttpServiceTest
com.company.component.server.controller.CredentialManagementControllerWebMvcTest
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testDeactivateCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGenerateCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialClientSecret
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialsHistory
com.company.component.server.controller.OrganizationControllerWebMvcTest
com.company.component.server.controller.OrganizationControllerWebMvcTest$testCreateOrganization
com.company.component.server.controller.OrganizationControllerWebMvcTest$testGetOrganization
com.company.component.server.datastore.credential.CredentialEntityTest
com.company.component.server.service.CredentialManagementServiceTest

Total Test Suites Not Shown: 23

Total Test Cases Not Shown: 55

Sample XML Report for Test Suite Not Shown in UI

Filename:

TEST-com.company.component.shared.helper.UriBuilderTest$HostWithPort.xml

Content:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Path part or params" tests="7" skipped="0" failures="0" errors="0" timestamp="2023-06-05T19:53:36" hostname="runner-xyz" time="0.029">
  <properties/>
  <testcase name="with dynamic path and no query params returns host + path param part" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.003"/>
  <testcase name="with dynamic path and no query params returns host + path param part" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.003"/>
  <testcase name="with no path and query param returns host + query param" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.003"/>
  <testcase name="with static path and no params returns host + path part" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.002"/>
  <testcase name="with dynamic path and without providing path value should throw due to missing path param value" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.002"/>
  <testcase name="with dynamic path and no query params returns host + path param part" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.004"/>
  <testcase name="with no path or params returns original host" classname="com.company.component.shared.helper.UriBuilderTest$HostWithPort" time="0.002"/>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

hey @ImFromMarzKa ! Thanks for the detailed report 😄 . From datadog-ci's perspective it seems the file are being uploaded correctly, and the xml files seem well formatted, so there might be an issue with our intake. I'm speaking with our backend team to figure this out. I'll keep you posted

hey @ImFromMarzKa the issue is that the timestamps from the reports are too old. Any report whose timestamp is older than 18 hours before the moment it is uploaded is discarded.

Are you maybe uploading old reports, or are the tests being run modifying the local time somehow? (some mock libraries do this)

hey @juan-fernandez , thank you so much for looking into it and for the quick responses. I think I know what the issue is.

We've been playing around with Gradle caching on our CI jobs. The unit test job is running with cache enabled. Since there are no code changes as part of this MR, it is pulling cached test results from a previous CI run, rather than rerunning all of our unit tests again. I have been debating whether this is good practice or if we should be running our tests every time. I'm leaning towards running every time anyways.

I will disable Gradle caching and test it out. I'll report back if it fixes my issue.

I wasn't aware of the 18 hour timestamp thing. I have spent a lot of time in your CI Visibility docs. If I may suggest adding a note about that to this troubleshooting guide, that may help someone out in the future!

Thanks again!

hey @ImFromMarzKa thanks for the context! This is useful info for us 😄 .

About the documentation, you're totally right: we'll update them to include this piece of info.

I have been debating whether this is good practice or if we should be running our tests every time. I'm leaning towards running every time anyways.

Just as an idea, another option is to skip the junit upload if there's a cache hit, since those results should already be in datadog (uploaded when the tests actually run).

I'll close this issue then and I'll let you know when docs are updated. Thank you!

@juan-fernandez that's a good point about skipping upload on cache hits.

I will discuss with my team and figure out the best approach. In this particular case, there are no test results in DataDog yet because this is the first we are integrating CI Test Visibility into our CI pipelines. So I need to get these results in there at least, but going forward we will make a call on the best approach.

Thanks again!

About the documentation, you're totally right: we'll update them to include this piece of info.

ℹ️ We're adding a troubleshooting section in https://github.com/DataDog/documentation/pull/18506 for this specific issue 😄

DataDog / datadog-ci