Closed ImFromMarzKa closed 1 year ago
hey @ImFromMarzKa ! Thanks for the detailed report š . From datadog-ci's perspective it seems the file are being uploaded correctly, and the xml files seem well formatted, so there might be an issue with our intake. I'm speaking with our backend team to figure this out. I'll keep you posted
hey @ImFromMarzKa the issue is that the timestamps from the reports are too old. Any report whose timestamp is older than 18 hours before the moment it is uploaded is discarded.
Are you maybe uploading old reports, or are the tests being run modifying the local time somehow? (some mock libraries do this)
hey @juan-fernandez , thank you so much for looking into it and for the quick responses. I think I know what the issue is.
We've been playing around with Gradle caching on our CI jobs. The unit test job is running with cache enabled. Since there are no code changes as part of this MR, it is pulling cached test results from a previous CI run, rather than rerunning all of our unit tests again. I have been debating whether this is good practice or if we should be running our tests every time. I'm leaning towards running every time anyways.
I will disable Gradle caching and test it out. I'll report back if it fixes my issue.
I wasn't aware of the 18 hour timestamp thing. I have spent a lot of time in your CI Visibility docs. If I may suggest adding a note about that to this troubleshooting guide, that may help someone out in the future!
Thanks again!
hey @ImFromMarzKa thanks for the context! This is useful info for us š .
About the documentation, you're totally right: we'll update them to include this piece of info.
I have been debating whether this is good practice or if we should be running our tests every time. I'm leaning towards running every time anyways.
Just as an idea, another option is to skip the junit upload if there's a cache hit, since those results should already be in datadog (uploaded when the tests actually run).
I'll close this issue then and I'll let you know when docs are updated. Thank you!
@juan-fernandez that's a good point about skipping upload on cache hits.
I will discuss with my team and figure out the best approach. In this particular case, there are no test results in DataDog yet because this is the first we are integrating CI Test Visibility into our CI pipelines. So I need to get these results in there at least, but going forward we will make a call on the best approach.
Thanks again!
About the documentation, you're totally right: we'll update them to include this piece of info.
ā¹ļø We're adding a troubleshooting section in https://github.com/DataDog/documentation/pull/18506 for this specific issue š
Summary
I am running
datadog-ci junit upload
on multiple test report directories using a single command. The log output from the command shows that all of the expected JUnit xml files were uploaded successfully.Expected Behavior: All test results are shown in DataDog CI Test UI after successful upload.
Actual Behavior: Test results from 23 out of 27 test suites are not shown in the DataDog CI Test UI, even though
datadog-ci
logs show successful upload of these test results.The reports for all of the missing test suites share the same directory structure (
**/build/test-results/test
). All missing test suites are executed by the same Gradle task (./gradlew test
). The other test suites that are shown in the UI are all executed by different Gradle tasks and their reports are in different directories.Details
CLI Version
CLI Installation
Installed via
yarn
.Initially,
datadog-ci
was installed on the Docker image used by the GitLab job like this:Once I ran into issues, I tried installing directly from the GitLab job script so that I had more control, like this:
CLI Command
Output from CLI Command
Test Suites Shown in UI
com.company.component.server.CharonServerProviderCDCT
com.company.component.client.CredentialManagementConsumerCDCT
com.company.component.client.OrganizationConsumerCDCT
com.company.component.server.OrganizationControllerIT
Total Test Suites Shown: 4
Total Test Cases Shown: 21
Sample XML Report for Test Suite Shown in UI
Filename:
TEST-com.company.component.client.CredentialManagementConsumerCDCT.xml
Content:
Test Suites Not Shown in UI
All test suites in directories matching:
**/build/test-results/test
com.company.component.api.model.IDTest
com.company.component.client.api.credential.CredentialManagementApiTest
com.company.component.client.api.organization.OrganizationApiTest
com.company.component.client.config.CharonServiceConfigurerTest
com.company.component.shared.config.service.ServiceConfigurerTest
com.company.component.shared.helper.UriBuilderTest$HostWithPort
com.company.component.shared.helper.UriBuilderTest$TestDifferentBaseURIs
com.company.component.shared.interceptor.Auth0ClientTokenInterceptorTest
com.company.component.shared.interceptor.BasicAuthInterceptorTest
com.company.component.shared.interceptor.RequestTelemetryInterceptorTest
com.company.component.shared.service.HttpBuilderServiceTest
com.company.component.shared.service.HttpServiceTest
com.company.component.server.controller.CredentialManagementControllerWebMvcTest
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testDeactivateCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGenerateCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredential
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialClientSecret
com.company.component.server.controller.CredentialManagementControllerWebMvcTest$testGetCredentialsHistory
com.company.component.server.controller.OrganizationControllerWebMvcTest
com.company.component.server.controller.OrganizationControllerWebMvcTest$testCreateOrganization
com.company.component.server.controller.OrganizationControllerWebMvcTest$testGetOrganization
com.company.component.server.datastore.credential.CredentialEntityTest
com.company.component.server.service.CredentialManagementServiceTest
Total Test Suites Not Shown: 23
Total Test Cases Not Shown: 55
Sample XML Report for Test Suite Not Shown in UI
Filename:
TEST-com.company.component.shared.helper.UriBuilderTest$HostWithPort.xml
Content: