Azure / azure-cosmos-db-emulator-docker

This repo serves as hub for managing issues, gathering feedback, and having discussions regarding the Cosmos DB Emulator Docker.
https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql
MIT License
153 stars 47 forks source link

Cosmos DB Linux Emulator fails to start on some Intel chips #45

Open milismsft opened 2 years ago

milismsft commented 2 years ago

Related to: https://github.com/actions/virtual-environments/issues/5036#issuecomment-1044270895

The Cosmos DB Linux Emulator fails to start on some Intel chips.

lscpu output: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian Address sizes: [46] CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Stepping: 7 CPU MHz: 2593.907 BogoMIPS: 87.81 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64KiB L1i cache: 64 KiB L2 cache: 2 MiB L3 cache: 35.8 MiB NUMA node0 CPU(s): 0,1 Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear /proc/cpuinfo content: /proc/cpuinfo

MicrosoftTeams-image (2)

niteshvijay1995 commented 3 months ago

@kntajus I was interested in knowing the emulator startup script. With your comment it looks like that you are not facing problem with Emulator startup but after successful startup some operation fails which causes test failure. If that's true, then we would need emulator logs to debug further.

niteshvijay1995 commented 3 months ago

Hi @niteshvijay1995,

We had a failure yesterday (in addition to a few more builds which were fine).

Machine information for failed build Example of error of failing test Diff of the machine information

@JonathanLydall Can you please confirm if the emulator startup is failing in failed build? Please share logs from docker. It should start with

This is an evaluation version.  There are [x] days left in the evaluation period.
Starting
Started 1/11 partitions
JonathanLydall commented 3 months ago

@niteshvijay1995, there is no text like that in the logs for the run, but only some of the tests against Cosmos DB are failing, not all.

Below is the start of the logs for the dotnet test execution. Of particular note you will see many requests succeed Like POSTs and GETs to /api/customer/<sub-path>, these endpoints are using Cosmos, meaning that the container is working for those tests, but then later fails.

Details

``` 2024-07-08T13:49:57.9960387Z ##[section]Starting: dotnet test 2024-07-08T13:49:57.9968976Z ============================================================================== 2024-07-08T13:49:57.9969129Z Task : .NET Core 2024-07-08T13:49:57.9969220Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command 2024-07-08T13:49:57.9969360Z Version : 2.242.0 2024-07-08T13:49:57.9969430Z Author : Microsoft Corporation 2024-07-08T13:49:57.9969543Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli 2024-07-08T13:49:57.9969671Z ============================================================================== 2024-07-08T13:49:58.6636365Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version. 2024-07-08T13:50:05.0632307Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/InternalTestModules/Intent.Modules.NET.InternalTestModules.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:05.0638143Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/Modules.Archived/Google PubSub/Publish.AspNetCore.GooglePubSub.TestApplication/Publish.AspNetCore.GooglePubSub.TestApplication.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:06.0468588Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/Modules.Archived/Google PubSub/Publish.CleanArch.GooglePubSub.TestApplication/Publish.CleanArch.GooglePubSub.TestApplication.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:07.7571689Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/Modules.Archived/Google PubSub/Subscribe.GooglePubSub.TestApplication/Subscribe.GooglePubSub.TestApplication.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:09.3437852Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/Modules/Intent.Modules.NET.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:11.0561647Z Test run for /home/vsts/work/1/s/Modules/Intent.Modules.VisualStudio.Projects.Tests/bin/Debug/net8.0/Intent.Modules.VisualStudio.Projects.Tests.dll (.NETCoreApp,Version=v8.0) 2024-07-08T13:50:11.1849442Z Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) 2024-07-08T13:50:11.1850304Z Copyright (c) Microsoft Corporation. All rights reserved. 2024-07-08T13:50:11.1922540Z 2024-07-08T13:50:11.4518071Z Starting test execution, please wait... 2024-07-08T13:50:11.5236195Z A total of 1 test files matched the specified pattern. 2024-07-08T13:50:15.9299083Z Results File: /home/vsts/work/_temp/_fv-az368-975_2024-07-08_13_50_15.trx 2024-07-08T13:50:15.9312202Z 2024-07-08T13:50:15.9390251Z Passed! - Failed: 0, Passed: 41, Skipped: 0, Total: 41, Duration: 595 ms - Intent.Modules.VisualStudio.Projects.Tests.dll (net8.0) 2024-07-08T13:50:18.5373669Z [command]/usr/bin/dotnet test /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.sln --logger trx --results-directory /home/vsts/work/_temp --no-build 2024-07-08T13:50:19.8734917Z Test run for /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/bin/Debug/net8.0/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.dll (.NETCoreApp,Version=v8.0) 2024-07-08T13:50:19.9902808Z Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) 2024-07-08T13:50:19.9909009Z Copyright (c) Microsoft Corporation. All rights reserved. 2024-07-08T13:50:20.0069168Z 2024-07-08T13:50:20.1646330Z Starting test execution, please wait... 2024-07-08T13:50:20.2173070Z A total of 1 test files matched the specified pattern. 2024-07-08T13:50:24.7635655Z [testcontainers.org 00:00:00.13] Connected to Docker: 2024-07-08T13:50:24.7636785Z Host: unix:///var/run/docker.sock 2024-07-08T13:50:24.7637861Z Server Version: 26.1.3 2024-07-08T13:50:24.7639823Z Kernel Version: 6.5.0-1022-azure 2024-07-08T13:50:24.7640686Z API Version: 1.45 2024-07-08T13:50:24.7641095Z Operating System: Ubuntu 22.04.4 LTS 2024-07-08T13:50:24.7641750Z Total Memory: 6.76 GB 2024-07-08T13:50:24.8229689Z [testcontainers.org 00:00:00.18] Searching Docker registry credential in Auths 2024-07-08T13:50:24.8235495Z [testcontainers.org 00:00:00.18] Searching Docker registry credential in CredHelpers 2024-07-08T13:50:24.8247134Z [testcontainers.org 00:00:00.19] Searching Docker registry credential in CredsStore 2024-07-08T13:50:24.8265303Z [testcontainers.org 00:00:00.19] Docker registry credential https://index.docker.io/v1/ found 2024-07-08T13:50:27.0913128Z [testcontainers.org 00:00:02.45] Docker image testcontainers/ryuk:0.6.0 created 2024-07-08T13:50:27.1920096Z [testcontainers.org 00:00:02.55] Docker container 2d2ae6048c84 created 2024-07-08T13:50:27.2553369Z [testcontainers.org 00:00:02.62] Start Docker container 2d2ae6048c84 2024-07-08T13:50:27.7320718Z [testcontainers.org 00:00:03.09] Wait for Docker container 2d2ae6048c84 to complete readiness checks 2024-07-08T13:50:27.7449018Z [testcontainers.org 00:00:03.11] Docker container 2d2ae6048c84 ready 2024-07-08T13:50:27.7704805Z [testcontainers.org 00:00:03.13] Searching Docker registry credential in Auths 2024-07-08T13:50:27.7708658Z [testcontainers.org 00:00:03.13] Searching Docker registry credential in Auths 2024-07-08T13:50:27.7714613Z [testcontainers.org 00:00:03.13] Searching Docker registry credential in CredHelpers 2024-07-08T13:50:27.7715750Z [testcontainers.org 00:00:03.13] Searching Docker registry credential in CredsStore 2024-07-08T13:50:27.7722812Z [testcontainers.org 00:00:03.13] Docker registry credential mcr.microsoft.com not found 2024-07-08T13:51:06.0218757Z [testcontainers.org 00:00:41.38] Docker image mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest created 2024-07-08T13:51:06.0434532Z [testcontainers.org 00:00:41.40] Docker container 0ff41da9c645 created 2024-07-08T13:51:06.0463657Z [testcontainers.org 00:00:41.41] Start Docker container 0ff41da9c645 2024-07-08T13:51:06.3693777Z [testcontainers.org 00:00:41.73] Wait for Docker container 0ff41da9c645 to complete readiness checks 2024-07-08T13:51:31.9831721Z [testcontainers.org 00:01:07.35] Docker container 0ff41da9c645 ready 2024-07-08T13:51:32.9248338Z [2024-07-08 13:51:32.880] [WRN] - No XML encryptor configured. Key {cb7bdc46-d99f-48b1-ae44-49544d0189b7} may be persisted to storage in unencrypted form. 2024-07-08T13:51:32.9929913Z [2024-07-08 13:51:32.989] [INF] - Initialized Scheduler Signaller of type: Quartz.Core.SchedulerSignalerImpl 2024-07-08T13:51:32.9930939Z [2024-07-08 13:51:32.990] [INF] - Quartz Scheduler created 2024-07-08T13:51:32.9931861Z [2024-07-08 13:51:32.990] [INF] - JobFactory set to: Quartz.Simpl.MicrosoftDependencyInjectionJobFactory 2024-07-08T13:51:32.9932721Z [2024-07-08 13:51:32.990] [INF] - RAMJobStore initialized. 2024-07-08T13:51:32.9933498Z [2024-07-08 13:51:32.991] [INF] - Quartz Scheduler 3.8.1.0 - 'QuartzScheduler' with instanceId 'NON_CLUSTERED' initialized 2024-07-08T13:51:33.0093126Z [2024-07-08 13:51:32.991] [INF] - Using thread pool 'Quartz.Simpl.DefaultThreadPool', size: 10 2024-07-08T13:51:33.0093831Z [2024-07-08 13:51:32.991] [INF] - Using job store 'Quartz.Simpl.RAMJobStore', supports persistence: False, clustered: False 2024-07-08T13:51:33.0094307Z [2024-07-08 13:51:33.007] [INF] - Adding 2 jobs, 2 triggers. 2024-07-08T13:51:33.0106762Z [2024-07-08 13:51:33.010] [INF] - Adding job: DEFAULT.CommandDelegateJob 2024-07-08T13:51:33.0337185Z [2024-07-08 13:51:33.032] [INF] - Adding job: DEFAULT.NewScheduledJob 2024-07-08T13:51:33.9687071Z [2024-07-08 13:51:33.964] [WRN] - Failed to determine the https port for redirect. 2024-07-08T13:51:34.2813210Z [2024-07-08 13:51:34.280] [INF] - Scheduler QuartzScheduler_$_NON_CLUSTERED started. 2024-07-08T13:51:38.6400875Z [2024-07-08 13:51:38.611] [ERR] - CleanArchitecture Request: Unhandled Exception for Request "UpdateEntityAfterEtagWasChangedByPreviousOperationTest" UpdateEntityAfterEtagWasChangedByPreviousOperationTest {} 2024-07-08T13:51:38.6408540Z Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: PreconditionFailed (412); Substatus: 0; ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334; Reason: ( 2024-07-08T13:51:38.6408958Z code : PreconditionFailed 2024-07-08T13:51:38.6411554Z message : Message: {"Errors":["One of the specified pre-condition is not met. Learn more: https:\/\/aka.ms\/CosmosDB\/sql\/errors\/precondition-failed"]} 2024-07-08T13:51:38.6413442Z ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: 2024-07-08T13:51:38.6414346Z RequestStartTime: 2024-07-08T13:51:38.5917398Z, RequestEndTime: 2024-07-08T13:51:38.5958371Z, Number of regions attempted:1 2024-07-08T13:51:38.6415460Z {"systemHistory":[{"dateUtc":"2024-07-08T13:51:33.4411179Z","cpu":100.000,"memory":3605024.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]} 2024-07-08T13:51:38.6426656Z RequestStart: 2024-07-08T13:51:38.5917398Z; ResponseTime: 2024-07-08T13:51:38.5958371Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 3, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 412, SubStatusCode: 0, RequestCharge: 1.24, ItemLSN: -1, SessionToken: -1#3, UsingLocalLSN: False, TransportException: null, BELatencyMs: 0.59, ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {"requestTimeline":[{"event": "Created", "startTimeUtc": "2024-07-08T13:51:38.5917398Z", "durationInMs": 0.1039},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2024-07-08T13:51:38.5918437Z", "durationInMs": 0.0027},{"event": "Pipelined", "startTimeUtc": "2024-07-08T13:51:38.5918464Z", "durationInMs": 0.3789},{"event": "Transit Time", "startTimeUtc": "2024-07-08T13:51:38.5922253Z", "durationInMs": 0.9483},{"event": "Received", "startTimeUtc": "2024-07-08T13:51:38.5931736Z", "durationInMs": 0.4703},{"event": "Completed", "startTimeUtc": "2024-07-08T13:51:38.5936439Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2024-07-08T13:51:38.5709247Z","lastSend":"2024-07-08T13:51:38.5709247Z","lastReceive":"2024-07-08T13:51:38.5832758Z"},"requestSizeInBytes":864,"requestBodySizeInBytes":264,"responseMetadataSizeInBytes":182,"responseBodySizeInBytes":134}; 2024-07-08T13:51:38.6429443Z ResourceType: Document, OperationType: Upsert 2024-07-08T13:51:38.6429667Z , SDK: Microsoft.Azure.Documents.Common/2.14.0 2024-07-08T13:51:38.6429854Z ); 2024-07-08T13:51:38.6430436Z at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode() 2024-07-08T13:51:38.6430759Z at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.ProcessMessage[T](ResponseMessage responseMessage, Func`2 createResponse) 2024-07-08T13:51:38.6431085Z at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateItemResponse[T](ResponseMessage responseMessage) 2024-07-08T13:51:38.6431483Z at Microsoft.Azure.Cosmos.ContainerCore.UpsertItemAsync[T](T item, ITrace trace, Nullable`1 partitionKey, ItemRequestOptions requestOptions, CancellationToken cancellationToken) 2024-07-08T13:51:38.6432139Z at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Func`2 openTelemetry, String operationName, RequestOptions requestOptions) 2024-07-08T13:51:38.6432789Z at Microsoft.Azure.Cosmos.ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func`2 task, Func`2 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel) 2024-07-08T13:51:38.6433329Z at Microsoft.Azure.CosmosRepository.DefaultRepository`1.UpdateAsync(TItem value, Boolean ignoreEtag, CancellationToken cancellationToken) 2024-07-08T13:51:38.6474424Z at AdvancedMappingCrud.Cosmos.Tests.Infrastructure.Repositories.CosmosDBRepositoryBase`3.<>c__DisplayClass8_0.<b__0>d.MoveNext() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Infrastructure/Repositories/CosmosDBRepositoryBase.cs:line 57 2024-07-08T13:51:38.6475706Z --- End of stack trace from previous location --- 2024-07-08T13:51:38.6476915Z at AdvancedMappingCrud.Cosmos.Tests.Infrastructure.Persistence.CosmosDBUnitOfWork.SaveChangesAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Infrastructure/Persistence/CosmosDBUnitOfWork.cs:line 46 2024-07-08T13:51:38.6477947Z at AdvancedMappingCrud.Cosmos.Tests.Application.Concurrency.UpdateEntityAfterEtagWasChangedByPreviousOperationTest.UpdateEntityAfterEtagWasChangedByPreviousOperationTestHandler.Handle(UpdateEntityAfterEtagWasChangedByPreviousOperationTest request, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Concurrency/UpdateEntityAfterEtagWasChangedByPreviousOperationTest/UpdateEntityAfterEtagWasChangedByPreviousOperationTestHandler.cs:line 70 2024-07-08T13:51:38.6478690Z at MediatR.Wrappers.RequestHandlerWrapperImpl`1.<>c__DisplayClass1_0.<g__Handler|0>d.MoveNext() 2024-07-08T13:51:38.6479112Z --- End of stack trace from previous location --- 2024-07-08T13:51:38.6479934Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.UnitOfWorkBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/UnitOfWorkBehaviour.cs:line 35 2024-07-08T13:51:38.6480620Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.ValidationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/ValidationBehaviour.cs:line 36 2024-07-08T13:51:38.6481328Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.AuthorizationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/AuthorizationBehaviour.cs:line 83 2024-07-08T13:51:38.6482055Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.PerformanceBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/PerformanceBehaviour.cs:line 35 2024-07-08T13:51:38.6483244Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.UnhandledExceptionBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/UnhandledExceptionBehaviour.cs:line 27 2024-07-08T13:51:38.6490604Z --- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(412, 0)":1}},"name":"UpsertItemAsync","start datetime":"2024-07-08T13:51:38.590Z","duration in milliseconds":18.2441,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T13:51:32.4496910Z","MachineId":"hashedMachineName:ff831680-7455-5626-5fcb-08efe054c3e9","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0736},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":16.9754,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":16.923,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T13:51:35.0864849Z","cpu":0.000,"memory":3582616.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":16.9068,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":16.8985,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":16.8436,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":16.8392,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":16.7582,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T13:51:38.5911577Z","DurationInMs":13.9362,"RequestUri":"https://localhost:32769/dbs/TestDb/colls/Container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"2e175277-50dd-4fc3-8a03-00cfb844d334","StatusCode":"PreconditionFailed","ReasonPhrase":"Precondition Failed"}]},"PointOperationStatisticsTraceDatum":{"Id":"PointOperationStatistics","ActivityId":"2e175277-50dd-4fc3-8a03-00cfb844d334","ResponseTimeUtc":"2024-07-08T13:51:38.6076082Z","StatusCode":412,"SubStatusCode":0,"RequestCharge":1.24,"RequestUri":"dbs/TestDb/colls/Container","ErrorMessage":null,"RequestSessionToken":null,"ResponseSessionToken":"0:-1#3","BELatencyInMs":"0.59"}}}]}]}]}]}]}]}]} 2024-07-08T13:51:38.6600866Z [2024-07-08 13:51:38.654] [ERR] - An unhandled exception has occurred while executing the request. 2024-07-08T13:51:38.6605684Z Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: PreconditionFailed (412); Substatus: 0; ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334; Reason: ( 2024-07-08T13:51:38.6606449Z code : PreconditionFailed 2024-07-08T13:51:38.6607281Z message : Message: {"Errors":["One of the specified pre-condition is not met. Learn more: https:\/\/aka.ms\/CosmosDB\/sql\/errors\/precondition-failed"]} 2024-07-08T13:51:38.6611975Z ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: 2024-07-08T13:51:38.6616915Z RequestStartTime: 2024-07-08T13:51:38.5917398Z, RequestEndTime: 2024-07-08T13:51:38.5958371Z, Number of regions attempted:1 2024-07-08T13:51:38.6618169Z {"systemHistory":[{"dateUtc":"2024-07-08T13:51:33.4411179Z","cpu":100.000,"memory":3605024.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]} 2024-07-08T13:51:38.6633993Z RequestStart: 2024-07-08T13:51:38.5917398Z; ResponseTime: 2024-07-08T13:51:38.5958371Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: 3, GlobalCommittedLsn: -1, PartitionKeyRangeId: 0, IsValid: True, StatusCode: 412, SubStatusCode: 0, RequestCharge: 1.24, ItemLSN: -1, SessionToken: -1#3, UsingLocalLSN: False, TransportException: null, BELatencyMs: 0.59, ActivityId: 2e175277-50dd-4fc3-8a03-00cfb844d334, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {"requestTimeline":[{"event": "Created", "startTimeUtc": "2024-07-08T13:51:38.5917398Z", "durationInMs": 0.1039},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2024-07-08T13:51:38.5918437Z", "durationInMs": 0.0027},{"event": "Pipelined", "startTimeUtc": "2024-07-08T13:51:38.5918464Z", "durationInMs": 0.3789},{"event": "Transit Time", "startTimeUtc": "2024-07-08T13:51:38.5922253Z", "durationInMs": 0.9483},{"event": "Received", "startTimeUtc": "2024-07-08T13:51:38.5931736Z", "durationInMs": 0.4703},{"event": "Completed", "startTimeUtc": "2024-07-08T13:51:38.5936439Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"False","callsPendingReceive":0,"lastSendAttempt":"2024-07-08T13:51:38.5709247Z","lastSend":"2024-07-08T13:51:38.5709247Z","lastReceive":"2024-07-08T13:51:38.5832758Z"},"requestSizeInBytes":864,"requestBodySizeInBytes":264,"responseMetadataSizeInBytes":182,"responseBodySizeInBytes":134}; 2024-07-08T13:51:38.6637554Z ResourceType: Document, OperationType: Upsert 2024-07-08T13:51:38.6638392Z , SDK: Microsoft.Azure.Documents.Common/2.14.0 2024-07-08T13:51:38.6638565Z ); 2024-07-08T13:51:38.6677775Z at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode() 2024-07-08T13:51:38.6678163Z at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.ProcessMessage[T](ResponseMessage responseMessage, Func`2 createResponse) 2024-07-08T13:51:38.6678516Z at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateItemResponse[T](ResponseMessage responseMessage) 2024-07-08T13:51:38.6678918Z at Microsoft.Azure.Cosmos.ContainerCore.UpsertItemAsync[T](T item, ITrace trace, Nullable`1 partitionKey, ItemRequestOptions requestOptions, CancellationToken cancellationToken) 2024-07-08T13:51:38.7365256Z at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func`2 task, Func`2 openTelemetry, String operationName, RequestOptions requestOptions) 2024-07-08T13:51:38.7366803Z at Microsoft.Azure.Cosmos.ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func`2 task, Func`2 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel) 2024-07-08T13:51:38.7367374Z at Microsoft.Azure.CosmosRepository.DefaultRepository`1.UpdateAsync(TItem value, Boolean ignoreEtag, CancellationToken cancellationToken) 2024-07-08T13:51:38.7368042Z at AdvancedMappingCrud.Cosmos.Tests.Infrastructure.Repositories.CosmosDBRepositoryBase`3.<>c__DisplayClass8_0.<b__0>d.MoveNext() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Infrastructure/Repositories/CosmosDBRepositoryBase.cs:line 57 2024-07-08T13:51:38.7370954Z --- End of stack trace from previous location --- 2024-07-08T13:51:38.7371481Z at AdvancedMappingCrud.Cosmos.Tests.Infrastructure.Persistence.CosmosDBUnitOfWork.SaveChangesAsync(CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Infrastructure/Persistence/CosmosDBUnitOfWork.cs:line 46 2024-07-08T13:51:38.7372401Z at AdvancedMappingCrud.Cosmos.Tests.Application.Concurrency.UpdateEntityAfterEtagWasChangedByPreviousOperationTest.UpdateEntityAfterEtagWasChangedByPreviousOperationTestHandler.Handle(UpdateEntityAfterEtagWasChangedByPreviousOperationTest request, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Concurrency/UpdateEntityAfterEtagWasChangedByPreviousOperationTest/UpdateEntityAfterEtagWasChangedByPreviousOperationTestHandler.cs:line 70 2024-07-08T13:51:38.7373386Z at MediatR.Wrappers.RequestHandlerWrapperImpl`1.<>c__DisplayClass1_0.<g__Handler|0>d.MoveNext() 2024-07-08T13:51:38.7373743Z --- End of stack trace from previous location --- 2024-07-08T13:51:38.7374240Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.UnitOfWorkBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/UnitOfWorkBehaviour.cs:line 35 2024-07-08T13:51:38.7374999Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.ValidationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/ValidationBehaviour.cs:line 36 2024-07-08T13:51:38.7375746Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.AuthorizationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/AuthorizationBehaviour.cs:line 83 2024-07-08T13:51:38.7376518Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.PerformanceBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/PerformanceBehaviour.cs:line 35 2024-07-08T13:51:38.7377289Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.UnhandledExceptionBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/UnhandledExceptionBehaviour.cs:line 27 2024-07-08T13:51:38.7378022Z at AdvancedMappingCrud.Cosmos.Tests.Api.Controllers.ConcurrencyController.UpdateEntityAfterEtagWasChangedByPreviousOperationTest(CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Api/Controllers/ConcurrencyController.cs:line 37 2024-07-08T13:51:38.7378473Z at lambda_method418(Closure, Object) 2024-07-08T13:51:38.7378854Z at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) 2024-07-08T13:51:38.7379359Z at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask) 2024-07-08T13:51:38.7379852Z at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-07-08T13:51:38.7380398Z at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) 2024-07-08T13:51:38.7380764Z at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2024-07-08T13:51:38.7381193Z at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-07-08T13:51:38.7381789Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|26_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-07-08T13:51:38.7382199Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ExceptionContextSealed context) 2024-07-08T13:51:38.7382546Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) 2024-07-08T13:51:38.7382975Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) 2024-07-08T13:51:38.7383403Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2024-07-08T13:51:38.7384195Z at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope) 2024-07-08T13:51:38.7384535Z at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) 2024-07-08T13:51:38.7384836Z at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) 2024-07-08T13:51:38.7385207Z at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task) 2024-07-08T13:51:38.7391450Z --- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(412, 0)":1}},"name":"UpsertItemAsync","start datetime":"2024-07-08T13:51:38.590Z","duration in milliseconds":18.2441,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T13:51:32.4496910Z","MachineId":"hashedMachineName:ff831680-7455-5626-5fcb-08efe054c3e9","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0736},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":16.9754,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":16.923,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T13:51:35.0864849Z","cpu":0.000,"memory":3582616.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":16.9068,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":16.8985,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":16.8436,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":16.8392,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":16.7582,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T13:51:38.5911577Z","DurationInMs":13.9362,"RequestUri":"https://localhost:32769/dbs/TestDb/colls/Container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"2e175277-50dd-4fc3-8a03-00cfb844d334","StatusCode":"PreconditionFailed","ReasonPhrase":"Precondition Failed"}]},"PointOperationStatisticsTraceDatum":{"Id":"PointOperationStatistics","ActivityId":"2e175277-50dd-4fc3-8a03-00cfb844d334","ResponseTimeUtc":"2024-07-08T13:51:38.6076082Z","StatusCode":412,"SubStatusCode":0,"RequestCharge":1.24,"RequestUri":"dbs/TestDb/colls/Container","ErrorMessage":null,"RequestSessionToken":null,"ResponseSessionToken":"0:-1#3","BELatencyInMs":"0.59"}}}]}]}]}]}]}]}]} 2024-07-08T13:51:38.7616527Z [2024-07-08 13:51:38.760] [ERR] - HTTP "PUT" "/api/concurrency" responded 500 in 4800.1913 ms 2024-07-08T13:51:39.1544376Z [2024-07-08 13:51:39.153] [INF] - HTTP "POST" "/api/customer" responded 201 in 119.0871 ms 2024-07-08T13:51:39.3517125Z [2024-07-08 13:51:39.351] [INF] - HTTP "GET" "/api/customer/c394af0d-ffd7-4c1b-9a98-6ee00da97c93" responded 200 in 186.4951 ms 2024-07-08T13:51:39.4341437Z [2024-07-08 13:51:39.432] [INF] - HTTP "POST" "/api/customer" responded 201 in 35.4794 ms 2024-07-08T13:51:39.5304934Z [2024-07-08 13:51:39.529] [INF] - HTTP "DELETE" "/api/customer/b769ce2d-1741-4045-a2b2-c5e609fca728" responded 200 in 92.6586 ms 2024-07-08T13:51:39.5817923Z [2024-07-08 13:51:39.580] [ERR] - CleanArchitecture Request: Unhandled Exception for Request "GetCustomerByIdQuery" GetCustomerByIdQuery {Id="b769ce2d-1741-4045-a2b2-c5e609fca728"} 2024-07-08T13:51:39.5821510Z AdvancedMappingCrud.Cosmos.Tests.Domain.Common.Exceptions.NotFoundException: Could not find Customer 'b769ce2d-1741-4045-a2b2-c5e609fca728' 2024-07-08T13:51:39.5824861Z at AdvancedMappingCrud.Cosmos.Tests.Application.Customers.GetCustomerById.GetCustomerByIdQueryHandler.Handle(GetCustomerByIdQuery request, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Customers/GetCustomerById/GetCustomerByIdQueryHandler.cs:line 34 2024-07-08T13:51:39.5830418Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.ValidationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/ValidationBehaviour.cs:line 36 2024-07-08T13:51:39.5836193Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.AuthorizationBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/AuthorizationBehaviour.cs:line 83 2024-07-08T13:51:39.5840071Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.PerformanceBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/PerformanceBehaviour.cs:line 35 2024-07-08T13:51:39.5859113Z at AdvancedMappingCrud.Cosmos.Tests.Application.Common.Behaviours.UnhandledExceptionBehaviour`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.Application/Common/Behaviours/UnhandledExceptionBehaviour.cs:line 27 2024-07-08T13:51:39.5920274Z [2024-07-08 13:51:39.591] [INF] - HTTP "GET" "/api/customer/b769ce2d-1741-4045-a2b2-c5e609fca728" responded 404 in 60.8433 ms 2024-07-08T13:51:39.6315470Z [2024-07-08 13:51:39.630] [INF] - HTTP "POST" "/api/customer" responded 201 in 19.8307 ms 2024-07-08T13:51:39.6594247Z [2024-07-08 13:51:39.658] [INF] - HTTP "GET" "/api/customer/ebfdf75d-6e53-4131-adab-2189961140b1" responded 200 in 18.7229 ms 2024-07-08T13:51:39.9493503Z [2024-07-08 13:51:39.948] [INF] - HTTP "GET" "/api/customer-by-name" responded 200 in 286.2991 ms 2024-07-08T13:51:39.9782294Z [2024-07-08 13:51:39.977] [INF] - HTTP "POST" "/api/customer" responded 201 in 15.7243 ms 2024-07-08T13:51:39.9895653Z [2024-07-08 13:51:39.989] [INF] - HTTP "GET" "/api/customer/58752db9-1622-4517-bafb-3a8d7c4d26c1" responded 200 in 9.2270 ms 2024-07-08T13:51:40.0250938Z [2024-07-08 13:51:40.024] [INF] - HTTP "POST" "/api/customer" responded 201 in 31.1829 ms 2024-07-08T13:51:40.0795399Z [2024-07-08 13:51:40.079] [INF] - HTTP "GET" "/api/customer" responded 200 in 50.4577 ms 2024-07-08T13:51:40.1121033Z [2024-07-08 13:51:40.111] [INF] - HTTP "POST" "/api/customer" responded 201 in 23.7069 ms 2024-07-08T13:51:40.1794909Z [2024-07-08 13:51:40.178] [INF] - HTTP "PUT" "/api/customer/d81ce1b2-65fb-4d81-a091-ee1421cbd79e" responded 204 in 51.3736 ms 2024-07-08T13:51:40.1919616Z [2024-07-08 13:51:40.191] [INF] - HTTP "GET" "/api/customer/d81ce1b2-65fb-4d81-a091-ee1421cbd79e" responded 200 in 11.8585 ms 2024-07-08T13:51:40.2202282Z [2024-07-08 13:51:40.219] [INF] - HTTP "POST" "/api/customer" responded 201 in 16.3880 ms 2024-07-08T13:51:40.3456241Z [2024-07-08 13:51:40.344] [INF] - HTTP "POST" "/api/product" responded 201 in 115.6627 ms 2024-07-08T13:51:40.5923629Z [2024-07-08 13:51:40.591] [INF] - HTTP "POST" "/api/order" responded 201 in 198.5899 ms 2024-07-08T13:51:40.6177082Z [2024-07-08 13:51:40.617] [INF] - HTTP "POST" "/api/product" responded 201 in 21.6037 ms 2024-07-08T13:51:40.7005372Z [2024-07-08 13:51:40.692] [INF] - HTTP "POST" "/api/order/order-item" responded 201 in 65.4794 ms 2024-07-08T13:51:40.7527350Z [2024-07-08 13:51:40.746] [INF] - HTTP "GET" "/api/order/order-item/3d931934-676f-45ac-98e9-f16d1578da66" responded 200 in 50.6651 ms 2024-07-08T13:51:40.7979120Z [2024-07-08 13:51:40.793] [INF] - HTTP "POST" "/api/customer" responded 201 in 29.5561 ms 2024-07-08T13:51:40.8322977Z [2024-07-08 13:51:40.831] [INF] - HTTP "POST" "/api/product" responded 201 in 30.6116 ms 2024-07-08T13:51:40.8627765Z [2024-07-08 13:51:40.858] [INF] - HTTP "POST" "/api/order" responded 201 in 19.5606 ms 2024-07-08T13:51:40.9458059Z [2024-07-08 13:51:40.944] [INF] - HTTP "GET" "/api/order/50c8e953-093a-46e6-b545-6bb626891b3b" responded 200 in 83.0956 ms 2024-07-08T13:51:40.9681644Z [2024-07-08 13:51:40.967] [INF] - HTTP "POST" "/api/customer" responded 201 in 12.3972 ms 2024-07-08T13:51:40.9921948Z [2024-07-08 13:51:40.991] [INF] - HTTP "POST" "/api/product" responded 201 in 20.7928 ms 2024-07-08T13:51:41.0360723Z [2024-07-08 13:51:41.035] [INF] - HTTP "POST" "/api/order" responded 201 in 36.9080 ms 2024-07-08T13:51:41.0601655Z [2024-07-08 13:51:41.059] [INF] - HTTP "POST" "/api/product" responded 201 in 21.9777 ms 2024-07-08T13:51:41.0880599Z [2024-07-08 13:51:41.087] [INF] - HTTP "POST" "/api/order/order-item" responded 201 in 24.0540 ms 2024-07-08T13:51:41.1305269Z [2024-07-08 13:51:41.129] [INF] - HTTP "DELETE" "/api/order/order-item/53cff706-5fe6-46c7-9f2e-4e4120847897" responded 200 in 37.0258 ms 2024-07-08T13:51:41.1672787Z [2024-07-08 13:51:41.166] [INF] - HTTP "GET" "/api/order/order-item/53cff706-5fe6-46c7-9f2e-4e4120847897" responded 404 in 35.2868 ms 2024-07-08T13:51:41.2015291Z [2024-07-08 13:51:41.200] [INF] - HTTP "POST" "/api/customer" responded 201 in 23.9361 ms 2024-07-08T13:51:41.2394646Z [2024-07-08 13:51:41.238] [INF] - HTTP "POST" "/api/product" responded 201 in 24.6960 ms 2024-07-08T13:51:41.2706115Z [2024-07-08 13:51:41.269] [INF] - HTTP "POST" "/api/order" responded 201 in 27.9936 ms 2024-07-08T13:51:41.3106291Z [2024-07-08 13:51:41.310] [INF] - HTTP "DELETE" "/api/order/2e87c4a3-c840-446a-863f-43f7c6d7c3f5" responded 200 in 37.4257 ms 2024-07-08T13:51:41.3694642Z [2024-07-08 13:51:41.369] [INF] - HTTP "GET" "/api/order/2e87c4a3-c840-446a-863f-43f7c6d7c3f5" responded 404 in 57.4975 ms 2024-07-08T13:51:41.4111197Z [2024-07-08 13:51:41.410] [INF] - HTTP "POST" "/api/customer" responded 201 in 29.4458 ms 2024-07-08T13:51:41.4320836Z [2024-07-08 13:51:41.431] [INF] - HTTP "POST" "/api/product" responded 201 in 16.2327 ms 2024-07-08T13:51:41.4610846Z [2024-07-08 13:51:41.460] [INF] - HTTP "POST" "/api/order" responded 201 in 22.2863 ms 2024-07-08T13:51:41.4792616Z [2024-07-08 13:51:41.478] [INF] - HTTP "GET" "/api/order/8814cc07-7624-438a-a447-f400c08231d2" responded 200 in 14.7256 ms 2024-07-08T13:51:41.5079422Z [2024-07-08 13:51:41.507] [INF] - HTTP "POST" "/api/customer" responded 201 in 18.9984 ms 2024-07-08T13:51:41.5452883Z [2024-07-08 13:51:41.544] [INF] - HTTP "POST" "/api/product" responded 201 in 34.0858 ms 2024-07-08T13:51:41.5949978Z [2024-07-08 13:51:41.594] [INF] - HTTP "POST" "/api/order" responded 201 in 41.6830 ms 2024-07-08T13:51:52.2478875Z [2024-07-08 13:51:52.246] [ERR] - CleanArchitecture Request: Unhandled Exception for Request "CreateProductCommand" CreateProductCommand {Name="Name353c0b0b-7a2e-4279-a919-36363aede2b7"} 2024-07-08T13:51:52.2517632Z Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195; Reason: ({"code":"RequestTimeout","message":"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\r\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \r\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\r\n{\"systemHistory\":[{\"dateUtc\":\"2024-07-08T13:51:33.4411179Z\",\"cpu\":100.000,\"memory\":3605024.000,\"threadInfo\":{\"isThreadStarving\":\"no info\",\"availableThreads\":32765,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":0},{\"dateUtc\":\"2024-07-08T13:51:43.4644654Z\",\"cpu\":100.000,\"memory\":3489228.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":446.7613,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2}]}\r\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\"requestTimeline\":[{\"event\": \"Created\", \"startTimeUtc\": \"2024-07-08T13:51:41.6114954Z\", \"durationInMs\": 5.1301},{\"event\": \"ChannelAcquisitionStarted\", \"startTimeUtc\": \"2024-07-08T13:51:41.6166255Z\", \"durationInMs\": 0.0053},{\"event\": \"Pipelined\", \"startTimeUtc\": \"2024-07-08T13:51:41.6166308Z\", \"durationInMs\": 0.8372},{\"event\": \"Transit Time\", \"startTimeUtc\": \"2024-07-08T13:51:41.6174680Z\", \"durationInMs\": 10587.9686},{\"event\": \"Failed\", \"startTimeUtc\": \"2024-07-08T13:51:52.2054366Z\", \"durationInMs\": 0}],\"serviceEndpointStats\":{\"inflightRequests\":1,\"openConnections\":1},\"connectionStats\":{\"waitforConnectionInit\":\"False\",\"callsPendingReceive\":0,\"lastSendAttempt\":\"2024-07-08T13:51:41.5555081Z\",\"lastSend\":\"2024-07-08T13:51:41.5555081Z\",\"lastReceive\":\"2024-07-08T13:51:41.5735847Z\"},\"requestSizeInBytes\":692,\"requestBodySizeInBytes\":137};\r\n ResourceType: Document, OperationType: Create\r\n, SDK: Microsoft.Azure.Documents.Common/2.14.0"} 2024-07-08T13:51:52.2522922Z RequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs; 2024-07-08T13:51:52.2612414Z RequestMethod: POST; 2024-07-08T13:51:52.2612633Z Header: Authorization Length: 86; 2024-07-08T13:51:52.2614969Z Header: x-ms-date Length: 29; 2024-07-08T13:51:52.2615926Z Header: x-ms-documentdb-partitionkey Length: 40; 2024-07-08T13:51:52.2616262Z Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1; 2024-07-08T13:51:52.2616627Z Header: x-ms-activity-id Length: 36; 2024-07-08T13:51:52.2616879Z Header: Cache-Control Length: 8; 2024-07-08T13:51:52.2617123Z Header: User-Agent Length: 79; 2024-07-08T13:51:52.2617366Z Header: x-ms-version Length: 10; 2024-07-08T13:51:52.2617539Z Header: Accept Length: 16; 2024-07-08T13:51:52.2617724Z Header: traceparent Length: 55; 2024-07-08T13:51:52.2618265Z , Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0); 2024-07-08T13:51:52.2618760Z at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request) 2024-07-08T13:51:52.2619207Z at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken) 2024-07-08T13:51:52.2619634Z at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2620009Z at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2620387Z at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2620758Z at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2636961Z --- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(408, 0)":1}},"name":"CreateItemAsync","start datetime":"2024-07-08T13:51:41.606Z","duration in milliseconds":10638.6776,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T13:51:32.4496910Z","MachineId":"hashedMachineName:ff831680-7455-5626-5fcb-08efe054c3e9","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0467},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":10638.3089,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":10638.2531,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T13:51:35.0864849Z","cpu":0.000,"memory":3582616.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T13:51:45.0959273Z","cpu":70.894,"memory":3486440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":922.4415,"availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":10638.2324,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":10638.2202,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":10637.9888,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":10637.5766,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":10631.9485,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T13:51:41.6068991Z","DurationInMs":10610.4762,"RequestUri":"https://localhost:32769/dbs/TestDb/colls/Container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"6bf62202-b027-4cc7-951d-b6cf7c2ab195","StatusCode":"RequestTimeout","ReasonPhrase":"Request timed out"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6bf62202-b027-4cc7-951d-b6cf7c2ab195","ResponseTimeUtc":"2024-07-08T13:51:52.2419595Z","StatusCode":408,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/TestDb/colls/Container","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\"code\":\"RequestTimeout\",\"message\":\"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\r\\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\r\\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\\r\\n{\\\"systemHistory\\\":[{\\\"dateUtc\\\":\\\"2024-07-08T13:51:33.4411179Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3605024.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"no info\\\",\\\"availableThreads\\\":32765,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":0},{\\\"dateUtc\\\":\\\"2024-07-08T13:51:43.4644654Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3489228.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":446.7613,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2}]}\\r\\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\\\"requestTimeline\\\":[{\\\"event\\\": \\\"Created\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6114954Z\\\", \\\"durationInMs\\\": 5.1301},{\\\"event\\\": \\\"ChannelAcquisitionStarted\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6166255Z\\\", \\\"durationInMs\\\": 0.0053},{\\\"event\\\": \\\"Pipelined\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6166308Z\\\", \\\"durationInMs\\\": 0.8372},{\\\"event\\\": \\\"Transit Time\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6174680Z\\\", \\\"durationInMs\\\": 10587.9686},{\\\"event\\\": \\\"Failed\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:52.2054366Z\\\", \\\"durationInMs\\\": 0}],\\\"serviceEndpointStats\\\":{\\\"inflightRequests\\\":1,\\\"openConnections\\\":1},\\\"connectionStats\\\":{\\\"waitforConnectionInit\\\":\\\"False\\\",\\\"callsPendingReceive\\\":0,\\\"lastSendAttempt\\\":\\\"2024-07-08T13:51:41.5555081Z\\\",\\\"lastSend\\\":\\\"2024-07-08T13:51:41.5555081Z\\\",\\\"lastReceive\\\":\\\"2024-07-08T13:51:41.5735847Z\\\"},\\\"requestSizeInBytes\\\":692,\\\"requestBodySizeInBytes\\\":137};\\r\\n ResourceType: Document, OperationType: Create\\r\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\"}\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\nRequestMethod: POST;\nHeader: Authorization Length: 86;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-documentdb-partitionkey Length: 40;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 79;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\nHeader: traceparent Length: 55;\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]}]} 2024-07-08T13:51:52.2650027Z [2024-07-08 13:51:52.259] [ERR] - An unhandled exception has occurred while executing the request. 2024-07-08T13:51:52.2656618Z Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195; Reason: ({"code":"RequestTimeout","message":"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\r\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \r\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\r\n{\"systemHistory\":[{\"dateUtc\":\"2024-07-08T13:51:33.4411179Z\",\"cpu\":100.000,\"memory\":3605024.000,\"threadInfo\":{\"isThreadStarving\":\"no info\",\"availableThreads\":32765,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":0},{\"dateUtc\":\"2024-07-08T13:51:43.4644654Z\",\"cpu\":100.000,\"memory\":3489228.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":446.7613,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2}]}\r\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\"requestTimeline\":[{\"event\": \"Created\", \"startTimeUtc\": \"2024-07-08T13:51:41.6114954Z\", \"durationInMs\": 5.1301},{\"event\": \"ChannelAcquisitionStarted\", \"startTimeUtc\": \"2024-07-08T13:51:41.6166255Z\", \"durationInMs\": 0.0053},{\"event\": \"Pipelined\", \"startTimeUtc\": \"2024-07-08T13:51:41.6166308Z\", \"durationInMs\": 0.8372},{\"event\": \"Transit Time\", \"startTimeUtc\": \"2024-07-08T13:51:41.6174680Z\", \"durationInMs\": 10587.9686},{\"event\": \"Failed\", \"startTimeUtc\": \"2024-07-08T13:51:52.2054366Z\", \"durationInMs\": 0}],\"serviceEndpointStats\":{\"inflightRequests\":1,\"openConnections\":1},\"connectionStats\":{\"waitforConnectionInit\":\"False\",\"callsPendingReceive\":0,\"lastSendAttempt\":\"2024-07-08T13:51:41.5555081Z\",\"lastSend\":\"2024-07-08T13:51:41.5555081Z\",\"lastReceive\":\"2024-07-08T13:51:41.5735847Z\"},\"requestSizeInBytes\":692,\"requestBodySizeInBytes\":137};\r\n ResourceType: Document, OperationType: Create\r\n, SDK: Microsoft.Azure.Documents.Common/2.14.0"} 2024-07-08T13:51:52.2660212Z RequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs; 2024-07-08T13:51:52.2660449Z RequestMethod: POST; 2024-07-08T13:51:52.2660618Z Header: Authorization Length: 86; 2024-07-08T13:51:52.2660887Z Header: x-ms-date Length: 29; 2024-07-08T13:51:52.2661171Z Header: x-ms-documentdb-partitionkey Length: 40; 2024-07-08T13:51:52.2661472Z Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1; 2024-07-08T13:51:52.2661732Z Header: x-ms-activity-id Length: 36; 2024-07-08T13:51:52.2661982Z Header: Cache-Control Length: 8; 2024-07-08T13:51:52.2662228Z Header: User-Agent Length: 79; 2024-07-08T13:51:52.2662471Z Header: x-ms-version Length: 10; 2024-07-08T13:51:52.2662649Z Header: Accept Length: 16; 2024-07-08T13:51:52.2662832Z Header: traceparent Length: 55; 2024-07-08T13:51:52.2663368Z , Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0); 2024-07-08T13:51:52.2663844Z at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request) 2024-07-08T13:51:52.2664424Z at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken) 2024-07-08T13:51:52.2664840Z at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2665230Z at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2667060Z at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2667852Z at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) 2024-07-08T13:51:52.2707768Z --- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(408, 0)":1}},"name":"CreateItemAsync","start datetime":"2024-07-08T13:51:41.606Z","duration in milliseconds":10638.6776,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T13:51:32.4496910Z","MachineId":"hashedMachineName:ff831680-7455-5626-5fcb-08efe054c3e9","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0467},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":10638.3089,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":10638.2531,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T13:51:35.0864849Z","cpu":0.000,"memory":3582616.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T13:51:45.0959273Z","cpu":70.894,"memory":3486440.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":922.4415,"availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":10638.2324,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":10638.2202,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":10637.9888,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":10637.5766,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":10631.9485,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T13:51:41.6068991Z","DurationInMs":10610.4762,"RequestUri":"https://localhost:32769/dbs/TestDb/colls/Container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"6bf62202-b027-4cc7-951d-b6cf7c2ab195","StatusCode":"RequestTimeout","ReasonPhrase":"Request timed out"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"6bf62202-b027-4cc7-951d-b6cf7c2ab195","ResponseTimeUtc":"2024-07-08T13:51:52.2419595Z","StatusCode":408,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/TestDb/colls/Container","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\"code\":\"RequestTimeout\",\"message\":\"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\r\\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\r\\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\\r\\n{\\\"systemHistory\\\":[{\\\"dateUtc\\\":\\\"2024-07-08T13:51:33.4411179Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3605024.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"no info\\\",\\\"availableThreads\\\":32765,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":0},{\\\"dateUtc\\\":\\\"2024-07-08T13:51:43.4644654Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3489228.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":446.7613,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2}]}\\r\\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\\\"requestTimeline\\\":[{\\\"event\\\": \\\"Created\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6114954Z\\\", \\\"durationInMs\\\": 5.1301},{\\\"event\\\": \\\"ChannelAcquisitionStarted\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6166255Z\\\", \\\"durationInMs\\\": 0.0053},{\\\"event\\\": \\\"Pipelined\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6166308Z\\\", \\\"durationInMs\\\": 0.8372},{\\\"event\\\": \\\"Transit Time\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:41.6174680Z\\\", \\\"durationInMs\\\": 10587.9686},{\\\"event\\\": \\\"Failed\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T13:51:52.2054366Z\\\", \\\"durationInMs\\\": 0}],\\\"serviceEndpointStats\\\":{\\\"inflightRequests\\\":1,\\\"openConnections\\\":1},\\\"connectionStats\\\":{\\\"waitforConnectionInit\\\":\\\"False\\\",\\\"callsPendingReceive\\\":0,\\\"lastSendAttempt\\\":\\\"2024-07-08T13:51:41.5555081Z\\\",\\\"lastSend\\\":\\\"2024-07-08T13:51:41.5555081Z\\\",\\\"lastReceive\\\":\\\"2024-07-08T13:51:41.5735847Z\\\"},\\\"requestSizeInBytes\\\":692,\\\"requestBodySizeInBytes\\\":137};\\r\\n ResourceType: Document, OperationType: Create\\r\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\"}\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\nRequestMethod: POST;\nHeader: Authorization Length: 86;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-documentdb-partitionkey Length: 40;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 79;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\nHeader: traceparent Length: 55;\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]}]} 2024-07-08T13:51:52.2756313Z [2024-07-08 13:51:52.274] [ERR] - HTTP "POST" "/api/product" responded 500 in 10669.1719 ms 2024-07-08T13:51:52.3263382Z [xUnit.net 00:01:31.49] AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.Tests.GetOrderOrderItemByIdTests.GetOrderOrderItemById_ShouldGetOrderOrderItemById [FAIL] 2024-07-08T13:51:52.3305417Z Failed AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.Tests.GetOrderOrderItemByIdTests.GetOrderOrderItemById_ShouldGetOrderOrderItemById [10 s] 2024-07-08T13:51:52.3306103Z Error Message: 2024-07-08T13:51:52.3306522Z AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.HttpClients.HttpClientRequestException : { 2024-07-08T13:51:52.3306914Z "type": "https://httpstatuses.io/500", 2024-07-08T13:51:52.3307750Z "title": "Internal Server Error", 2024-07-08T13:51:52.3310985Z "status": 500, 2024-07-08T13:51:52.3347842Z "detail": "Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195; Reason: ({\u0022code\u0022:\u0022RequestTimeout\u0022,\u0022message\u0022:\u0022Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\r\\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\r\\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\\r\\n{\\\u0022systemHistory\\\u0022:[{\\\u0022dateUtc\\\u0022:\\\u00222024-07-08T13:51:33.4411179Z\\\u0022,\\\u0022cpu\\\u0022:100.000,\\\u0022memory\\\u0022:3605024.000,\\\u0022threadInfo\\\u0022:{\\\u0022isThreadStarving\\\u0022:\\\u0022no info\\\u0022,\\\u0022availableThreads\\\u0022:32765,\\\u0022minThreads\\\u0022:2,\\\u0022maxThreads\\\u0022:32767},\\\u0022numberOfOpenTcpConnection\\\u0022:0},{\\\u0022dateUtc\\\u0022:\\\u00222024-07-08T13:51:43.4644654Z\\\u0022,\\\u0022cpu\\\u0022:100.000,\\\u0022memory\\\u0022:3489228.000,\\\u0022threadInfo\\\u0022:{\\\u0022isThreadStarving\\\u0022:\\\u0022False\\\u0022,\\\u0022threadWaitIntervalInMs\\\u0022:446.7613,\\\u0022availableThreads\\\u0022:32766,\\\u0022minThreads\\\u0022:2,\\\u0022maxThreads\\\u0022:32767},\\\u0022numberOfOpenTcpConnection\\\u0022:2}]}\\r\\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -\u003E 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\\\u0022requestTimeline\\\u0022:[{\\\u0022event\\\u0022: \\\u0022Created\\\u0022, \\\u0022startTimeUtc\\\u0022: \\\u00222024-07-08T13:51:41.6114954Z\\\u0022, \\\u0022durationInMs\\\u0022: 5.1301},{\\\u0022event\\\u0022: \\\u0022ChannelAcquisitionStarted\\\u0022, \\\u0022startTimeUtc\\\u0022: \\\u00222024-07-08T13:51:41.6166255Z\\\u0022, \\\u0022durationInMs\\\u0022: 0.0053},{\\\u0022event\\\u0022: \\\u0022Pipelined\\\u0022, \\\u0022startTimeUtc\\\u0022: \\\u00222024-07-08T13:51:41.6166308Z\\\u0022, \\\u0022durationInMs\\\u0022: 0.8372},{\\\u0022event\\\u0022: \\\u0022Transit Time\\\u0022, \\\u0022startTimeUtc\\\u0022: \\\u00222024-07-08T13:51:41.6174680Z\\\u0022, \\\u0022durationInMs\\\u0022: 10587.9686},{\\\u0022event\\\u0022: \\\u0022Failed\\\u0022, \\\u0022startTimeUtc\\\u0022: \\\u00222024-07-08T13:51:52.2054366Z\\\u0022, \\\u0022durationInMs\\\u0022: 0}],\\\u0022serviceEndpointStats\\\u0022:{\\\u0022inflightRequests\\\u0022:1,\\\u0022openConnections\\\u0022:1},\\\u0022connectionStats\\\u0022:{\\\u0022waitforConnectionInit\\\u0022:\\\u0022False\\\u0022,\\\u0022callsPendingReceive\\\u0022:0,\\\u0022lastSendAttempt\\\u0022:\\\u00222024-07-08T13:51:41.5555081Z\\\u0022,\\\u0022lastSend\\\u0022:\\\u00222024-07-08T13:51:41.5555081Z\\\u0022,\\\u0022lastReceive\\\u0022:\\\u00222024-07-08T13:51:41.5735847Z\\\u0022},\\\u0022requestSizeInBytes\\\u0022:692,\\\u0022requestBodySizeInBytes\\\u0022:137};\\r\\n ResourceType: Document, OperationType: Create\\r\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\u0022}\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\nRequestMethod: POST;\nHeader: Authorization Length: 86;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-documentdb-partitionkey Length: 40;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 79;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\nHeader: traceparent Length: 55;\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0);\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)\n--- Cosmos Diagnostics ---{\u0022Summary\u0022:{\u0022GatewayCalls\u0022:{\u0022(408, 0)\u0022:1}},\u0022name\u0022:\u0022CreateItemAsync\u0022,\u0022start datetime\u0022:\u00222024-07-08T13:51:41.606Z\u0022,\u0022duration in milliseconds\u0022:10638.6776,\u0022data\u0022:{\u0022Client Configuration\u0022:{\u0022Client Created Time Utc\u0022:\u00222024-07-08T13:51:32.4496910Z\u0022,\u0022MachineId\u0022:\u0022hashedMachineName:ff831680-7455-5626-5fcb-08efe054c3e9\u0022,\u0022NumberOfClientsCreated\u0022:1,\u0022NumberOfActiveClients\u0022:1,\u0022ConnectionMode\u0022:\u0022Gateway\u0022,\u0022User Agent\u0022:\u0022cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|\u0022,\u0022ConnectionConfig\u0022:{\u0022gw\u0022:\u0022(cps:50, urto:6, p:False, httpf: True)\u0022,\u0022rntbd\u0022:\u0022(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)\u0022,\u0022other\u0022:\u0022(ed:False, be:False)\u0022},\u0022ConsistencyConfig\u0022:\u0022(consistency: NotSet, prgns:[], apprgn: )\u0022,\u0022ProcessorCount\u0022:2}},\u0022children\u0022:[{\u0022name\u0022:\u0022ItemSerialize\u0022,\u0022duration in milliseconds\u0022:0.0467},{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler\u0022,\u0022duration in milliseconds\u0022:10638.3089,\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler\u0022,\u0022duration in milliseconds\u0022:10638.2531,\u0022data\u0022:{\u0022System Info\u0022:{\u0022systemHistory\u0022:[{\u0022dateUtc\u0022:\u00222024-07-08T13:51:35.0864849Z\u0022,\u0022cpu\u0022:0.000,\u0022memory\u0022:3582616.000,\u0022threadInfo\u0022:{\u0022isThreadStarving\u0022:\u0022no info\u0022,\u0022availableThreads\u0022:32764,\u0022minThreads\u0022:2,\u0022maxThreads\u0022:32767},\u0022numberOfOpenTcpConnection\u0022:0},{\u0022dateUtc\u0022:\u00222024-07-08T13:51:45.0959273Z\u0022,\u0022cpu\u0022:70.894,\u0022memory\u0022:3486440.000,\u0022threadInfo\u0022:{\u0022isThreadStarving\u0022:\u0022False\u0022,\u0022threadWaitIntervalInMs\u0022:922.4415,\u0022availableThreads\u0022:32764,\u0022minThreads\u0022:2,\u0022maxThreads\u0022:32767},\u0022numberOfOpenTcpConnection\u0022:0}]}},\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.TelemetryHandler\u0022,\u0022duration in milliseconds\u0022:10638.2324,\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.RetryHandler\u0022,\u0022duration in milliseconds\u0022:10638.2202,\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.RouterHandler\u0022,\u0022duration in milliseconds\u0022:10637.9888,\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.Handlers.TransportHandler\u0022,\u0022duration in milliseconds\u0022:10637.5766,\u0022children\u0022:[{\u0022name\u0022:\u0022Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request\u0022,\u0022duration in milliseconds\u0022:10631.9485,\u0022data\u0022:{\u0022Client Side Request Stats\u0022:{\u0022Id\u0022:\u0022AggregatedClientSideRequestStatistics\u0022,\u0022ContactedReplicas\u0022:[],\u0022RegionsContacted\u0022:[],\u0022FailedReplicas\u0022:[],\u0022AddressResolutionStatistics\u0022:[],\u0022StoreResponseStatistics\u0022:[],\u0022HttpResponseStats\u0022:[{\u0022StartTimeUTC\u0022:\u00222024-07-08T13:51:41.6068991Z\u0022,\u0022DurationInMs\u0022:10610.4762,\u0022RequestUri\u0022:\u0022https://localhost:32769/dbs/TestDb/colls/Container/docs\u0022,\u0022ResourceType\u0022:\u0022Document\u0022,\u0022HttpMethod\u0022:\u0022POST\u0022,\u0022ActivityId\u0022:\u00226bf62202-b027-4cc7-951d-b6cf7c2ab195\u0022,\u0022StatusCode\u0022:\u0022RequestTimeout\u0022,\u0022ReasonPhrase\u0022:\u0022Request timed out\u0022}]},\u0022Point Operation Statistics\u0022:{\u0022Id\u0022:\u0022PointOperationStatistics\u0022,\u0022ActivityId\u0022:\u00226bf62202-b027-4cc7-951d-b6cf7c2ab195\u0022,\u0022ResponseTimeUtc\u0022:\u00222024-07-08T13:51:52.2419595Z\u0022,\u0022StatusCode\u0022:408,\u0022SubStatusCode\u0022:0,\u0022RequestCharge\u0022:0,\u0022RequestUri\u0022:\u0022dbs/TestDb/colls/Container\u0022,\u0022ErrorMessage\u0022:\u0022Microsoft.Azure.Documents.DocumentClientException: {\\\u0022code\\\u0022:\\\u0022RequestTimeout\\\u0022,\\\u0022message\\\u0022:\\\u0022Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\\\r\\\\nActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\\\r\\\\nRequestStartTime: 2024-07-08T13:51:41.6114954Z, RequestEndTime: 2024-07-08T13:51:52.2122674Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\\u0022systemHistory\\\\\\\u0022:[{\\\\\\\u0022dateUtc\\\\\\\u0022:\\\\\\\u00222024-07-08T13:51:33.4411179Z\\\\\\\u0022,\\\\\\\u0022cpu\\\\\\\u0022:100.000,\\\\\\\u0022memory\\\\\\\u0022:3605024.000,\\\\\\\u0022threadInfo\\\\\\\u0022:{\\\\\\\u0022isThreadStarving\\\\\\\u0022:\\\\\\\u0022no info\\\\\\\u0022,\\\\\\\u0022availableThreads\\\\\\\u0022:32765,\\\\\\\u0022minThreads\\\\\\\u0022:2,\\\\\\\u0022maxThreads\\\\\\\u0022:32767},\\\\\\\u0022numberOfOpenTcpConnection\\\\\\\u0022:0},{\\\\\\\u0022dateUtc\\\\\\\u0022:\\\\\\\u00222024-07-08T13:51:43.4644654Z\\\\\\\u0022,\\\\\\\u0022cpu\\\\\\\u0022:100.000,\\\\\\\u0022memory\\\\\\\u0022:3489228.000,\\\\\\\u0022threadInfo\\\\\\\u0022:{\\\\\\\u0022isThreadStarving\\\\\\\u0022:\\\\\\\u0022False\\\\\\\u0022,\\\\\\\u0022threadWaitIntervalInMs\\\\\\\u0022:446.7613,\\\\\\\u0022availableThreads\\\\\\\u0022:32766,\\\\\\\u0022minThreads\\\\\\\u0022:2,\\\\\\\u0022maxThreads\\\\\\\u0022:32767},\\\\\\\u0022numberOfOpenTcpConnection\\\\\\\u0022:2}]}\\\\r\\\\nRequestStart: 2024-07-08T13:51:41.6114954Z; ResponseTime: 2024-07-08T13:51:52.2122674Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T13:51:52.2040964Z, activity ID: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:42845 -\u003E 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: 6bf62202-b027-4cc7-951d-b6cf7c2ab195, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 13:51:37)], TransportRequestTimeline: {\\\\\\\u0022requestTimeline\\\\\\\u0022:[{\\\\\\\u0022event\\\\\\\u0022: \\\\\\\u0022Created\\\\\\\u0022, \\\\\\\u0022startTimeUtc\\\\\\\u0022: \\\\\\\u00222024-07-08T13:51:41.6114954Z\\\\\\\u0022, \\\\\\\u0022durationInMs\\\\\\\u0022: 5.1301},{\\\\\\\u0022event\\\\\\\u0022: \\\\\\\u0022ChannelAcquisitionStarted\\\\\\\u0022, \\\\\\\u0022startTimeUtc\\\\\\\u0022: \\\\\\\u00222024-07-08T13:51:41.6166255Z\\\\\\\u0022, \\\\\\\u0022durationInMs\\\\\\\u0022: 0.0053},{\\\\\\\u0022event\\\\\\\u0022: \\\\\\\u0022Pipelined\\\\\\\u0022, \\\\\\\u0022startTimeUtc\\\\\\\u0022: \\\\\\\u00222024-07-08T13:51:41.6166308Z\\\\\\\u0022, \\\\\\\u0022durationInMs\\\\\\\u0022: 0.8372},{\\\\\\\u0022event\\\\\\\u0022: \\\\\\\u0022Transit Time\\\\\\\u0022, \\\\\\\u0022startTimeUtc\\\\\\\u0022: \\\\\\\u00222024-07-08T13:51:41.6174680Z\\\\\\\u0022, \\\\\\\u0022durationInMs\\\\\\\u0022: 10587.9686},{\\\\\\\u0022event\\\\\\\u0022: \\\\\\\u0022Failed\\\\\\\u0022, \\\\\\\u0022startTimeUtc\\\\\\\u0022: \\\\\\\u00222024-07-08T13:51:52.2054366Z\\\\\\\u0022, \\\\\\\u0022durationInMs\\\\\\\u0022: 0}],\\\\\\\u0022serviceEndpointStats\\\\\\\u0022:{\\\\\\\u0022inflightRequests\\\\\\\u0022:1,\\\\\\\u0022openConnections\\\\\\\u0022:1},\\\\\\\u0022connectionStats\\\\\\\u0022:{\\\\\\\u0022waitforConnectionInit\\\\\\\u0022:\\\\\\\u0022False\\\\\\\u0022,\\\\\\\u0022callsPendingReceive\\\\\\\u0022:0,\\\\\\\u0022lastSendAttempt\\\\\\\u0022:\\\\\\\u00222024-07-08T13:51:41.5555081Z\\\\\\\u0022,\\\\\\\u0022lastSend\\\\\\\u0022:\\\\\\\u00222024-07-08T13:51:41.5555081Z\\\\\\\u0022,\\\\\\\u0022lastReceive\\\\\\\u0022:\\\\\\\u00222024-07-08T13:51:41.5735847Z\\\\\\\u0022},\\\\\\\u0022requestSizeInBytes\\\\\\\u0022:692,\\\\\\\u0022requestBodySizeInBytes\\\\\\\u0022:137};\\\\r\\\\n ResourceType: Document, OperationType: Create\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\\\u0022}\\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\\nRequestMethod: POST;\\nHeader: Authorization Length: 86;\\nHeader: x-ms-date Length: 29;\\nHeader: x-ms-documentdb-partitionkey Length: 40;\\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\\nHeader: x-ms-activity-id Length: 36;\\nHeader: Cache-Control Length: 8;\\nHeader: User-Agent Length: 79;\\nHeader: x-ms-version Length: 10;\\nHeader: Accept Length: 16;\\nHeader: traceparent Length: 55;\\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0\\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)\u0022,\u0022RequestSessionToken\u0022:null,\u0022ResponseSessionToken\u0022:null,\u0022BELatencyInMs\u0022:null}}}]}]}]}]}]}]}]}", 2024-07-08T13:51:52.3367484Z "traceId": "00-e49e978b9057ee1f74aaad5334039405-8846d09ebca46574-00" 2024-07-08T13:51:52.3367699Z } 2024-07-08T13:51:52.3367852Z Stack Trace: 2024-07-08T13:51:52.3368331Z at AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.HttpClients.Products.ProductsHttpClient.CreateProductAsync(CreateProductCommand command, CancellationToken cancellationToken) in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/HttpClients/Products/ProductsHttpClient.cs:line 43 2024-07-08T13:51:52.3369011Z at AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.TestDataFactory.CreateProduct() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/TestDataFactory.cs:line 88 2024-07-08T13:51:52.3369583Z at AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.TestDataFactory.CreateOrderItemDependencies() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/TestDataFactory.cs:line 67 2024-07-08T13:51:52.3370134Z at AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.TestDataFactory.CreateOrderItem() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/TestDataFactory.cs:line 73 2024-07-08T13:51:52.3370770Z at AdvancedMappingCrud.Cosmos.Tests.IntegrationTests.Tests.GetOrderOrderItemByIdTests.GetOrderOrderItemById_ShouldGetOrderOrderItemById() in /home/vsts/work/1/s/Tests/AdvancedMappingCrud.Cosmos.Tests/AdvancedMappingCrud.Cosmos.Tests.IntegrationTests/Tests/Orders/GetOrderOrderItemByIdTests.cs:line 31 2024-07-08T13:51:52.3371311Z --- End of stack trace from previous location --- ```

kntajus commented 3 months ago

@niteshvijay1995 I'm looking in to how to grab the emulator logs for you. In my local testing I'm seeing nothing of interest in there. For a full run where all the tests are passing, it's literally just giving me:

2024-07-09T14:48:25.077797514Z This is an evaluation version.  There are [96] days left in the evaluation period.
2024-07-09T14:48:29.014322440Z Starting
2024-07-09T14:48:52.741548345Z Starting 1/11 partitions
...
2024-07-09T14:49:16.981054047Z Started 11/11 partitions
2024-07-09T14:49:16.981083958Z Started

Is there some kind of flag or environment variable I should be setting to tell the emulator to output more verbose logs?

razvangoga commented 3 months ago

@kntajus i think the logs should be accessible via a docker logs cli command - if i remember correctly the service containers in a devops pipeline run in the docker instance bootstrapped for that pipeline run

kntajus commented 3 months ago

Thanks @razvangoga, given that I'm using Testcontainers I've found a way to access the container logs directly from the test code anyway.

@JonathanLydall I don't know if this might be useful for you too, looks like you're using .NET - you can call GetLogsAsync on the instance representing the docker container that Testcontainers provides for you at the end of the test/run.

razvangoga commented 3 months ago

@kntajus cool!

genuinely curious if anything comes from them

the issue is hardware related and the behaviour is in line with how AzDevops assign agent VMs: based on the region of the AzDevops tenant you will get agents VMs in the same Azure region or in a fallback one ( docs )

it explains why some people (like me based in Germany where agents come from Germany or France azure regions) get only fails, while some others have more success (different azure zone => different hardware).

i mean we can also wait a couple of years more until MS upgrades all it's datacenteres from these pesky Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz 😁

in any case i'll drink a beer each day @niteshvijay1995 doesn't ghost the repo like @sajeetharan did the last time around

Blackbaud-JasonBodnar commented 3 months ago

I find it quite amusing that this issue has a Feature label. The emulator is supposed to work on Linux. There's nothing about only certain Linux agents in ADO. This is a bug not a feature.