search

Azure / azure-sdk-for-cpp

This repository is for active development of the Azure SDK for C++. For consumers of the SDK we recommend visiting our versioned developer docs at https://azure.github.io/azure-sdk-for-cpp.
MIT License
177 stars 127 forks source link

Intermittent BlobContainerClientTest.Tags test failure due to HTTP status code 202 and `FindBlobsByTags` not returning the expected tags #5788

Open ahsonkhan opened 2 months ago

ahsonkhan commented 2 months ago

From an unrelated PR: https://github.com/Azure/azure-sdk-for-cpp/pull/5772

On ValidateLive Win2022_x86_with_unit_test_winHttp https://dev.azure.com/azure-sdk/internal/_build/results?buildId=3954909&view=logs&j=1cbef409-5ce5-5885-71aa-60f628bf5123&t=614bdaae-21d9-5723-b317-b9e0e4565c8b&l=434709

Test: https://github.com/Azure/azure-sdk-for-cpp/blob/1143b6bf4dc1eb3ffb9cf5ba4cb7ebb22f5ff073/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp#L846

2024-07-11T22:26:16.9327118Z 118: [2024-07-11T22:26:16.9307174Z T: 4604] INFO  : HTTP status code 200 won't be retried.
2024-07-11T22:26:17.9426083Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(954): error: Expected: (numPages1) > (2), actual: 1 vs 2
2024-07-11T22:26:17.9428592Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(955): error: Expected: (numPages2) > (2), actual: 1 vs 2
2024-07-11T22:26:17.9430137Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(956): error: Expected equality of these values:
2024-07-11T22:26:17.9431451Z 118:   findResults.size()
2024-07-11T22:26:17.9432319Z 118:     Which is: 0
2024-07-11T22:26:17.9433120Z 118:   blobNames.size()
2024-07-11T22:26:17.9433879Z 118:     Which is: 6
2024-07-11T22:26:17.9434978Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(957): error: Expected equality of these values:
2024-07-11T22:26:17.9435737Z 118:   findResults2.size()
2024-07-11T22:26:17.9436448Z 118:     Which is: 0
2024-07-11T22:26:17.9437421Z 118:   blobNames.size()
2024-07-11T22:26:17.9438407Z 118:     Which is: 6
2024-07-11T22:26:17.9439666Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(961): error: Expected equality of these values:
2024-07-11T22:26:17.9440254Z 118:   blobNames
2024-07-11T22:26:17.9440757Z 118:     Which is: { "blobQdSWwHWvd0", "blobQdSWwHWvd00", "blobQdSWwHWvd01", "blobQdSWwHWvd02", "blobQdSWwHWvd03", "blobQdSWwHWvd04" }
2024-07-11T22:26:17.9441428Z 118:   findResults
2024-07-11T22:26:17.9441804Z 118:     Which is: {}
2024-07-11T22:26:17.9442446Z 118: D:\a\_work\1\s\sdk\storage\azure-storage-blobs\test\ut\blob_container_client_test.cpp(962): error: Expected equality of these values:
2024-07-11T22:26:17.9442942Z 118:   blobNames
2024-07-11T22:26:17.9443442Z 118:     Which is: { "blobQdSWwHWvd0", "blobQdSWwHWvd00", "blobQdSWwHWvd01", "blobQdSWwHWvd02", "blobQdSWwHWvd03", "blobQdSWwHWvd04" }
2024-07-11T22:26:17.9443911Z 118:   findResults2
2024-07-11T22:26:17.9444291Z 118:     Which is: {}

See detailed logs attached ValidateLive Win2022_x86_with_unit_test_winHttp-BlobContainerClientTest.Tags.txt

ahsonkhan commented 2 months ago

This test has failed twice, recently (in the last 2 weeks). https://dev.azure.com/azure-sdk/internal/_test/analytics?definitionId=1770&contextType=build

image

Jinming-Hu commented 2 months ago

This is server-side issue. We can increase this value to reduce the possibility of failing. However, it will increase the running time of this testcase. https://github.com/Azure/azure-sdk-for-cpp/blob/9ccd206ff88f1550ecf94139806085b69a341cd2/sdk/storage/azure-storage-blobs/test/ut/blob_container_client_test.cpp#L907

ahsonkhan commented 2 months ago

Is the server-side issue, something that we need to request to be fixed? Is there some way we increase the determinism/reliability of the test and it's expectations without increasing the test run time?

Having flaky tests, that intermittently fail, ultimately defeats the purpose of tests passing/failing, and increases the signal-to-noise ratio when a test does "really" fail which requires investigation to fix a product/test bug.