Cosmos DB Linux Emulator fails to start on some Intel chips

[milismsft]

Related to: https://github.com/actions/virtual-environments/issues/5036#issuecomment-1044270895

The Cosmos DB Linux Emulator fails to start on some Intel chips.

lscpu output: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian Address sizes: [46] CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Stepping: 7 CPU MHz: 2593.907 BogoMIPS: 87.81 Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64KiB L1i cache: 64 KiB L2 cache: 2 MiB L3 cache: 35.8 MiB NUMA node0 CPU(s): 0,1 Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear /proc/cpuinfo content: /proc/cpuinfo

[aressler38]

Is there a way to add a constraint on the Azure Pipeline to use the CPU model that works? I am hitting this issue in Azure DevOps Pipelines, and I always get model 85, which always fails. I have tried specifying "ubuntu-latest" "ubuntu-20.04" and "ubuntu-18.04" but none have worked.

The below CPU also fails.

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
stepping    : 7
microcode   : 0xffffffff
cpu MHz     : 2593.905
cache size  : 36608 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 21
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips    : 5187.81
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

This one with ubuntu-18.04 did work:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz
stepping    : 4
microcode   : 0xffffffff
cpu MHz     : 2095.078
cache size  : 36608 KB
physical id : 0
siblings    : 2
core id     : 0
cpu cores   : 2
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 21
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
bugs        : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips    : 4190.15
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

[rrr-michael-aquilina]

I am also facing this issue. Are there any work arounds or has any progress been made?

[soenneker]

@rrr-michael-aquilina The workaround that worked for me was moving to the Cosmos emulator (powershell) that's baked in the windows pipeline.

High traffic times can cause the emulator to start slowly, > 5min. I had to make some modifications to it's timeout and such but it's been pretty stable since.

Definitely worth having than not using the emulator at all in the DevOps pipeline.

[christian-be]

Also faced this issue using the Linux Docker image. Cost me a day of investigating network issues just to find the out the container is immediately shutting down. Using ubuntu-18.04 as suggested in the other Github ticket worked for me, but a fix for 20-04 would be great.

[waszak]

Ubuntu agent 18.04 is getting depracated so the issue needs to be fixed before that day. https://github.com/actions/runner-images/issues/6002

[mj-rittermann]

We are seeing the exact same problem. Running fine on Azure DevOps agents running ubuntu-18.04 but fails on ubuntu-20.04 and ubuntu-22.04. Would someone please look in to a fix on this, as the ubuntu-18.04 image are beeing deprecated on 4/1/2023 as @waszak mentioned: https://github.com/actions/runner-images/issues/6002

[eddumelendez]

I run a test today and it works on 20.04, see https://github.com/eddumelendez/testcontainers-cosmodb-gha-test/actions/runs/3153248862/jobs/5129495371

Can someone else confirm?

[nogic1008]

@eddumelendez Not yet. I'm using Cosmos DB container as Service container on GitHub Actions, but "Connection Refused" error still occurs.

https://github.com/ddradar/ddradar/pull/1002 https://github.com/ddradar/ddradar/actions/runs/3155822253/jobs/5134896830

[eddumelendez]

I think it is flaky, ran two more times and the first failed but the last one succeeded

[mmoayyed]

Yes it's flaky. I continue to see random failures as well.

[waszak]

@milismsft do you have any updates on this issue?

[dankarmyy]

+1 for working on ubuntu 20/22

We run as part of integration testing - only starts (sometimes) on ubuntu 18. But anything higher it just hangs at the "Starting" message in the container logs forever.

Our devs use docker-compose stack for local dependencies which includes cosmosdb, so we would like to just spin up the same stack in ado pipelines.

Ubuntu 18 deprecation date was pushed back to April '23 so we have a bit more time...

[LevYas]

Very actual during the current un-scheduled brownout for 18! 20 doesn't work.

[waszak]

This repo doesn't look like is active so I posted question here.

https://learn.microsoft.com/en-us/answers/questions/1057083/cosmos-db-linux-emulator-doesn39t-work-on-some-int.html

[Meandron]

Any news on this?

[waszak]

Someone asked again today but all we got is the same answer.

We don't have a public facing ETA we can share for now, but we will share on Azure updates when this will be available.

[asos-gurpreetsingh]

while we wait on this, is there a workaround ? I am using windows agent to get around this problem but the emulator for windows agent randomly takes too long to start

[soenneker]

while we wait on this, is there a workaround ? I am using windows agent to get around this problem but the emulator for windows agent randomly takes too long to start

Utilize a retry with your PowerShell task

[kiview]

Since today the next scheduled brown-out of the Ubuntu 18.04 GHA runners happened and we are getting closer to EOL for those runners, any updates or workarounds, especially for GHA users?

[eli-fin]

We are blocked on this issue too. Any update?

[DSpirit]

We are blocked on this issue too. Any update?

Our current workaround is, to self-host agents with a different chipset. See my answer here: #56

[eli-fin]

We are blocked on this issue too. Any update?

Our current workaround is, to self-host agents with a different chipset. See my answer here: #56

Our org has strict policies regarding self-hosted agents, so not as straight forward. But thanks.

[asos-gurpreetsingh]

We are blocked on this issue too. Any update?

Our current workaround is, to self-host agents with a different chipset. See my answer here: #56

Our org has strict policies regarding self-hosted agents, so not as straight forward. But thanks.

We moved this job to a windows agent and rest are on ubuntu to get around this issue.

[eli-fin]

We moved this job to a windows agent and rest are on ubuntu to get around this issue.

OK. How can your code running on ubuntu access the db running on the windows agent?

[soenneker]

@milismsft any update here? Ubuntu 18.04 isn't available anymore so it essentially prevents us from using Linux agents..

[aomegax]

any news?

[guibranco]

Any updates on this?

[sajeetharan]

Hi it is not supported yet, but we are actively exploring options to support this.

[tpischke]

This is blocking our project from running Integration Tests on github with CosmosDB, so I hope this will be fixed soon. No update since Oct 12 is not particularly encouraging. Since this involves a very basic use case for two flagship products, I would hope this would get prompt attention.

[razvangoga]

@sajeetharan will the new version you mentioned in #79 here also fix this issue? (we're trying to use the emulator as part of our integration tests in an Azure Devops Pipeline)

[rrr-michael-aquilina]

@sajeetharan - Can you provide an update on your previous comment? It would be great to use this emulator for end to end integration tests via Azure DevOps!

[sajeetharan]

hi @rrr-michael-aquilina @razvangoga @tpischke just to confirm, are you referring to the Mac OS intel chip support or https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/79 ?

[LevYas]

I struggle the most with the fact that the Emulator doesn't work in GitHub Actions, like stated in the first post.

[razvangoga]

Same for me but in Azure Devops on ubuntu 20 or 22

On Wed, Feb 7, 2024 at 10:20 Lev Yastrebov @.***> wrote:

I struggle the most with the fact that the Emulator doesn't work in GitHub Actions, like stated in the first post.

— Reply to this email directly, view it on GitHub https://github.com/Azure/azure-cosmos-db-emulator-docker/issues/45#issuecomment-1931611647, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIETZSVR2ZDKQHYQB5BYYTYSNBPVAVCNFSM5OYZCENKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJTGE3DCMJWGQ3Q . You are receiving this because you were mentioned.Message ID: @.***>

[AButler]

hi @rrr-michael-aquilina @razvangoga @tpischke just to confirm, are you referring to the Mac OS intel chip support or #79 ?

Hi @sajeetharan,

I think what most people want is to be able to run the Cosmos emulator in a pipeline/action using Azure DevOps or GitHub. As the Linux runners that both GitHub/DevOps use do not work with the emulator because of the issue described in the first post, we've either had to switch to a Windows runner or built a custom runner.

[rrr-michael-aquilina]

@sajeetharan - I was referring to the original post. For example, ubuntu-latest azure devops agent.

[joelsteventurner]

Was really looking forward to getting some CI / CD integration tests going with a containerized Cosmos Emulator in our Azure dev-ops pipelines but it seems to be failing intermittently.

This has been great and works very consistently when running on my desktop in Visual Studio.

I have since included these tests as part of a CI/CD pipeline and the tests are failing intermittently (roughly 50% of the time I'd say). This failure seems to be because the container is being stopped.. for some unknown reason. (sometimes...)

The failed runs seem to happen quickly e.g. within 3-5 seconds of the container starting, I see a Stop Docker container in the log messages there is no "reason" or error I can see for why the container has been stopped. Below of samples of failed and successful runs and my container configuration.

Discussion on Testcontainers

[JonathanLydall]

@sajeetharan, it's essentially been two years now since this issue was opened, two years since people reported that it's blocking them from being able to use Microsoft hosted Linux runners on Azure DevOps Pipelines to be able to run integration tests against Cosmos DB using the docker emulator.

I feel really let down by Microsoft here, no explanation has been offered as to why this hasn't been fixed yet, nor has a practical workaround been offered, for example all the following are not options which would work for us:

• Switching to Windows agents: Other containers we use for other integration tests can't run under Windows agents, and it's important for us that our tests in general run on Linux to spot other potential cross-platform issues.
• Self hosted agent: The reason we pay for Microsoft hosted agents is so that we don't have the burden of managing our own systems, updating software, etc.
• Running against production Cosmos DB: We would need to alter our test suite for this to work and incur additional fees (on top what we're already paying for Microsoft hosted build agents!)

It feels like this blocking issue is not being prioritised by Microsoft appropriately, between the Azure DevOps Pipelines and the Cosmos DB Emulator teams, can this please get the attention it deserves.

[jon-signal]

Building on @JonathanLydall's excellent summary: perhaps it would be productive to change the title of this issue to "Cosmos DB does not work with Microsoft-hosted Linux runners on Azure DevOps Pipelines."

[DoxxableSetanta]

Any update on this? This is ridiculous.

[soenneker]

Building on @JonathanLydall's excellent summary: perhaps it would be productive to change the title of this issue to "Cosmos DB does not work with Microsoft-hosted Linux runners on Azure DevOps Pipelines."

This would be incorrect though. It doesn't need to be Microsoft-hosted, and it doesn't need to be on Azure DevOps. It happens on most mainstream flavors of linux, on any platform (namely, GitHub)

[diegosasw]

I've been running some xUnit samples using the CosmosDb emulator both with Testcontainers and FluentDocker.

See my repo https://github.com/diegosasw/cosmosdb-sample/tree/634f6044ba7fe384bcb2d34ebef3f653ef85be0b

They seem to run properly locally and in CI/CD in GitHub actions using ubuntu latest.

Am I missing anything? has this issue finally been resolved with latest emulator? or it depends on how the docker container is spined up?

name: Test CosmosDb pipeline on Linux

on:
  push:
    branches:
      - main

jobs:
  unit_tests:
    name: Run .NET unit tests (Linux)
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run .NET tests
        run: dotnet test

[JonathanLydall]

Doesn't seem to be fixed, I just tested on a Microsoft hosted ubuntu-latest agent on Azure Pipelines and it failed on the first try.

[niteshvijay1995]

@JonathanLydall Please provide the yaml file that you are using to run emulator. Also, if possible, please help us with machine info of agent that is used to run pipeline.

Following task can help in Printing Machine Info -

- task: PowerShell@2
    name: PrintMachineInformation
    inputs:
      targetType: 'inline'
      script: |
        echo "CPU Information:"
        lscpu
        echo "Memory Information:"
        free -h
        echo "Disk Usage:"
        df -h
        echo "Operating System Information:"
        uname -a
        echo "Network Configuration:"
        ip addr show
        echo "Docker Version:"
        docker --version
        echo "Docker Info:"
        docker info
      pwsh: true

[JonathanLydall]

Hi @niteshvijay1995,

Here is the YAML file: https://github.com/IntentArchitect/Intent.Modules.NET/blob/1cde41ccb3c8498ccba4a38d5060414eacdea762/azure-pipelines.yml#L137

Just a note that the task YAML you pasted above has incorrect white spacing, it should be:

- task: PowerShell@2
  name: 'PrintMachineInformation'
  inputs:
    targetType: 'inline'
    pwsh: true
    script: |
      echo "CPU Information:"
      lscpu
      echo "Memory Information:"
      free -h
      echo "Disk Usage:"
      df -h
      echo "Operating System Information:"
      uname -a
      echo "Network Configuration:"
      ip addr show
      echo "Docker Version:"
      docker --version
      echo "Docker Info:"
      docker info

I updated the Pipeline now to add the information task and also to re-enable Cosmos DB container image tests again.

Since enabling it this morning 2 of 2 runs have succeeded, however, as this issue was intermittent before (I guess depending on which underlying hardware Azure Pipelines happened to use for a particular run), I wouldn't consider it solved until a few more days of testing.

In the meantime, here is the PrintMachineInformation output of the two successful runs:

Run 1

``` 2024-07-08T06:22:36.6098004Z ##[section]Starting: PrintMachineInformation 2024-07-08T06:22:36.6390812Z ============================================================================== 2024-07-08T06:22:36.6391888Z Task : PowerShell 2024-07-08T06:22:36.6392427Z Description : Run a PowerShell script on Linux, macOS, or Windows 2024-07-08T06:22:36.6392586Z Version : 2.239.1 2024-07-08T06:22:36.6392878Z Author : Microsoft Corporation 2024-07-08T06:22:36.6393200Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell 2024-07-08T06:22:36.6393338Z ============================================================================== 2024-07-08T06:22:37.1905094Z Generating script. 2024-07-08T06:22:37.1939819Z ========================== Starting Command Output =========================== 2024-07-08T06:22:37.2116637Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/ec24e82d-e7fa-4de5-95c5-306e9ec7ba53.ps1' 2024-07-08T06:22:38.7292677Z CPU Information: 2024-07-08T06:22:38.8100451Z Architecture: x86_64 2024-07-08T06:22:38.8101683Z CPU op-mode(s): 32-bit, 64-bit 2024-07-08T06:22:38.8102239Z Address sizes: 46 bits physical, 48 bits virtual 2024-07-08T06:22:38.8102739Z Byte Order: Little Endian 2024-07-08T06:22:38.8103533Z CPU(s): 2 2024-07-08T06:22:38.8103917Z On-line CPU(s) list: 0,1 2024-07-08T06:22:38.8104310Z Vendor ID: GenuineIntel 2024-07-08T06:22:38.8105214Z Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz 2024-07-08T06:22:38.8105781Z CPU family: 6 2024-07-08T06:22:38.8105949Z Model: 79 2024-07-08T06:22:38.8106098Z Thread(s) per core: 1 2024-07-08T06:22:38.8106323Z Core(s) per socket: 2 2024-07-08T06:22:38.8108347Z Socket(s): 1 2024-07-08T06:22:38.8108878Z Stepping: 1 2024-07-08T06:22:38.8109287Z BogoMIPS: 4589.37 2024-07-08T06:22:38.8109858Z Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear 2024-07-08T06:22:38.8110694Z Hypervisor vendor: Microsoft 2024-07-08T06:22:38.8111477Z Virtualization type: full 2024-07-08T06:22:38.8112173Z L1d cache: 64 KiB (2 instances) 2024-07-08T06:22:38.8112722Z L1i cache: 64 KiB (2 instances) 2024-07-08T06:22:38.8113228Z L2 cache: 512 KiB (2 instances) 2024-07-08T06:22:38.8113659Z L3 cache: 50 MiB (1 instance) 2024-07-08T06:22:38.8114000Z NUMA node(s): 1 2024-07-08T06:22:38.8114373Z NUMA node0 CPU(s): 0,1 2024-07-08T06:22:38.8222979Z Vulnerability Gather data sampling: Not affected 2024-07-08T06:22:38.8223232Z Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported 2024-07-08T06:22:38.8223470Z Vulnerability L1tf: Mitigation; PTE Inversion 2024-07-08T06:22:38.8223704Z Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T06:22:38.8223960Z Vulnerability Meltdown: Mitigation; PTI 2024-07-08T06:22:38.8224214Z Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown 2024-07-08T06:22:38.8224442Z Vulnerability Retbleed: Not affected 2024-07-08T06:22:38.8224949Z Vulnerability Spec rstack overflow: Not affected 2024-07-08T06:22:38.8225477Z Vulnerability Spec store bypass: Vulnerable 2024-07-08T06:22:38.8225915Z Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization 2024-07-08T06:22:38.8226412Z Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline 2024-07-08T06:22:38.8226649Z Vulnerability Srbds: Not affected 2024-07-08T06:22:38.8226838Z Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T06:22:38.8227032Z Memory Information: 2024-07-08T06:22:38.8279638Z total used free shared buff/cache available 2024-07-08T06:22:38.8280177Z Mem: 6.8Gi 672Mi 4.6Gi 35Mi 1.5Gi 5.8Gi 2024-07-08T06:22:38.8280343Z Swap: 4.0Gi 0B 4.0Gi 2024-07-08T06:22:38.8320585Z Disk Usage: 2024-07-08T06:22:38.8399195Z Filesystem Size Used Avail Use% Mounted on 2024-07-08T06:22:38.8400047Z /dev/root 73G 52G 21G 72% / 2024-07-08T06:22:38.8400362Z tmpfs 3.4G 172K 3.4G 1% /dev/shm 2024-07-08T06:22:38.8400536Z tmpfs 1.4G 1.1M 1.4G 1% /run 2024-07-08T06:22:38.8400687Z tmpfs 5.0M 0 5.0M 0% /run/lock 2024-07-08T06:22:38.8400857Z /dev/sdb15 105M 6.1M 99M 6% /boot/efi 2024-07-08T06:22:38.8401011Z /dev/sda1 14G 4.1G 9.0G 31% /mnt 2024-07-08T06:22:38.8401187Z tmpfs 693M 12K 693M 1% /run/user/1001 2024-07-08T06:22:38.8411521Z Operating System Information: 2024-07-08T06:22:38.8532344Z Linux fv-az367-389 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-07-08T06:22:38.8538630Z Network Configuration: 2024-07-08T06:22:38.8594195Z 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-08T06:22:38.8594500Z link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-08T06:22:38.8594792Z inet 127.0.0.1/8 scope host lo 2024-07-08T06:22:38.8594948Z valid_lft forever preferred_lft forever 2024-07-08T06:22:38.8595121Z inet6 ::1/128 scope host 2024-07-08T06:22:38.8595272Z valid_lft forever preferred_lft forever 2024-07-08T06:22:38.8595478Z 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-08T06:22:38.8598128Z link/ether 00:22:48:9a:99:5f brd ff:ff:ff:ff:ff:ff 2024-07-08T06:22:38.8598496Z inet 10.1.0.5/16 metric 100 brd 10.1.255.255 scope global eth0 2024-07-08T06:22:38.8598716Z valid_lft forever preferred_lft forever 2024-07-08T06:22:38.8598913Z inet6 fe80::222:48ff:fe9a:995f/64 scope link 2024-07-08T06:22:38.8599125Z valid_lft forever preferred_lft forever 2024-07-08T06:22:38.8599542Z 3: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-07-08T06:22:38.8600156Z link/ether 02:42:0e:68:7c:0b brd ff:ff:ff:ff:ff:ff 2024-07-08T06:22:38.8600416Z inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-07-08T06:22:38.8600607Z valid_lft forever preferred_lft forever 2024-07-08T06:22:38.8600944Z Docker Version: 2024-07-08T06:22:38.9996316Z Docker version 26.1.3, build b72abbb 2024-07-08T06:22:39.0012838Z Docker Info: 2024-07-08T06:22:39.4671472Z Client: Docker Engine - Community 2024-07-08T06:22:39.4672561Z Version: 26.1.3 2024-07-08T06:22:39.4673664Z Context: default 2024-07-08T06:22:39.4674018Z Debug Mode: false 2024-07-08T06:22:39.4674606Z Plugins: 2024-07-08T06:22:39.4676710Z buildx: Docker Buildx (Docker Inc.) 2024-07-08T06:22:39.4677090Z Version: v0.15.1 2024-07-08T06:22:39.4677527Z Path: /usr/libexec/docker/cli-plugins/docker-buildx 2024-07-08T06:22:39.4678155Z compose: Docker Compose (Docker Inc.) 2024-07-08T06:22:39.4678605Z Version: v2.27.1 2024-07-08T06:22:39.4679086Z Path: /usr/libexec/docker/cli-plugins/docker-compose 2024-07-08T06:22:39.4680073Z 2024-07-08T06:22:39.4680363Z Server: 2024-07-08T06:22:39.4680964Z Containers: 0 2024-07-08T06:22:39.4683623Z Running: 0 2024-07-08T06:22:39.4684288Z Paused: 0 2024-07-08T06:22:39.4684651Z Stopped: 0 2024-07-08T06:22:39.4684934Z Images: 14 2024-07-08T06:22:39.4685292Z Server Version: 26.1.3 2024-07-08T06:22:39.4685710Z Storage Driver: overlay2 2024-07-08T06:22:39.4686142Z Backing Filesystem: extfs 2024-07-08T06:22:39.4686669Z Supports d_type: true 2024-07-08T06:22:39.4687037Z Using metacopy: false 2024-07-08T06:22:39.4687421Z Native Overlay Diff: false 2024-07-08T06:22:39.4687806Z userxattr: false 2024-07-08T06:22:39.4688208Z Logging Driver: json-file 2024-07-08T06:22:39.4688612Z Cgroup Driver: cgroupfs 2024-07-08T06:22:39.4688982Z Cgroup Version: 2 2024-07-08T06:22:39.4689387Z Plugins: 2024-07-08T06:22:39.4689746Z Volume: local 2024-07-08T06:22:39.4690088Z Network: bridge host ipvlan macvlan null overlay 2024-07-08T06:22:39.4690597Z Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog 2024-07-08T06:22:39.4691048Z Swarm: inactive 2024-07-08T06:22:39.4691446Z Runtimes: io.containerd.runc.v2 runc 2024-07-08T06:22:39.4692160Z Default Runtime: runc 2024-07-08T06:22:39.4692886Z Init Binary: docker-init 2024-07-08T06:22:39.4693525Z containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e 2024-07-08T06:22:39.4694180Z runc version: v1.1.13-0-g58aa920 2024-07-08T06:22:39.4694546Z init version: de40ad0 2024-07-08T06:22:39.4694853Z Security Options: 2024-07-08T06:22:39.4695155Z apparmor 2024-07-08T06:22:39.4695446Z seccomp 2024-07-08T06:22:39.4695804Z Profile: builtin 2024-07-08T06:22:39.4696139Z cgroupns 2024-07-08T06:22:39.4696611Z Kernel Version: 6.5.0-1022-azure 2024-07-08T06:22:39.4697037Z Operating System: Ubuntu 22.04.4 LTS 2024-07-08T06:22:39.4697422Z OSType: linux 2024-07-08T06:22:39.4697801Z Architecture: x86_64 2024-07-08T06:22:39.4698183Z CPUs: 2 2024-07-08T06:22:39.4698545Z Total Memory: 6.759GiB 2024-07-08T06:22:39.4698968Z Name: fv-az367-389 2024-07-08T06:22:39.4699461Z ID: 2eba0d88-daa1-420e-94b5-971f98899c95 2024-07-08T06:22:39.4700088Z Docker Root Dir: /var/lib/docker 2024-07-08T06:22:39.4700510Z Debug Mode: false 2024-07-08T06:22:39.4700902Z Username: githubactions 2024-07-08T06:22:39.4701501Z Experimental: false 2024-07-08T06:22:39.4701825Z Insecure Registries: 2024-07-08T06:22:39.4702136Z 127.0.0.0/8 2024-07-08T06:22:39.4702509Z Live Restore Enabled: false 2024-07-08T06:22:39.4702766Z 2024-07-08T06:22:39.6321188Z 2024-07-08T06:22:39.6434901Z ##[section]Finishing: PrintMachineInformation ```

Run 2

``` 2024-07-08T07:08:56.4934603Z ##[section]Starting: PrintMachineInformation 2024-07-08T07:08:56.5173503Z ============================================================================== 2024-07-08T07:08:56.5174061Z Task : PowerShell 2024-07-08T07:08:56.5174271Z Description : Run a PowerShell script on Linux, macOS, or Windows 2024-07-08T07:08:56.5174395Z Version : 2.239.1 2024-07-08T07:08:56.5174586Z Author : Microsoft Corporation 2024-07-08T07:08:56.5174788Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell 2024-07-08T07:08:56.5174904Z ============================================================================== 2024-07-08T07:08:56.9038540Z Generating script. 2024-07-08T07:08:56.9071228Z ========================== Starting Command Output =========================== 2024-07-08T07:08:56.9249699Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/30803d64-d9f4-4ee7-9759-abf0260d7a5a.ps1' 2024-07-08T07:08:58.1801503Z CPU Information: 2024-07-08T07:08:58.2366720Z Architecture: x86_64 2024-07-08T07:08:58.2367623Z CPU op-mode(s): 32-bit, 64-bit 2024-07-08T07:08:58.2369036Z Address sizes: 46 bits physical, 48 bits virtual 2024-07-08T07:08:58.2369505Z Byte Order: Little Endian 2024-07-08T07:08:58.2369898Z CPU(s): 2 2024-07-08T07:08:58.2370299Z On-line CPU(s) list: 0,1 2024-07-08T07:08:58.2370624Z Vendor ID: GenuineIntel 2024-07-08T07:08:58.2371096Z Model name: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz 2024-07-08T07:08:58.2371876Z CPU family: 6 2024-07-08T07:08:58.2372056Z Model: 106 2024-07-08T07:08:58.2372195Z Thread(s) per core: 1 2024-07-08T07:08:58.2372350Z Core(s) per socket: 2 2024-07-08T07:08:58.2372474Z Socket(s): 1 2024-07-08T07:08:58.2372620Z Stepping: 6 2024-07-08T07:08:58.2372753Z BogoMIPS: 5586.87 2024-07-08T07:08:58.2373200Z Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear 2024-07-08T07:08:58.2373661Z Hypervisor vendor: Microsoft 2024-07-08T07:08:58.2373809Z Virtualization type: full 2024-07-08T07:08:58.2373951Z L1d cache: 96 KiB (2 instances) 2024-07-08T07:08:58.2374109Z L1i cache: 64 KiB (2 instances) 2024-07-08T07:08:58.2374256Z L2 cache: 2.5 MiB (2 instances) 2024-07-08T07:08:58.2374412Z L3 cache: 48 MiB (1 instance) 2024-07-08T07:08:58.2374548Z NUMA node(s): 1 2024-07-08T07:08:58.2374674Z NUMA node0 CPU(s): 0,1 2024-07-08T07:08:58.2374848Z Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status 2024-07-08T07:08:58.2375049Z Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported 2024-07-08T07:08:58.2375229Z Vulnerability L1tf: Mitigation; PTE Inversion 2024-07-08T07:08:58.2375410Z Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T07:08:58.2375643Z Vulnerability Meltdown: Mitigation; PTI 2024-07-08T07:08:58.2375938Z Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown 2024-07-08T07:08:58.2376398Z Vulnerability Retbleed: Not affected 2024-07-08T07:08:58.2376567Z Vulnerability Spec rstack overflow: Not affected 2024-07-08T07:08:58.2377073Z Vulnerability Spec store bypass: Vulnerable 2024-07-08T07:08:58.2377270Z Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization 2024-07-08T07:08:58.2377767Z Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline 2024-07-08T07:08:58.2377973Z Vulnerability Srbds: Not affected 2024-07-08T07:08:58.2378169Z Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T07:08:58.2384385Z Memory Information: 2024-07-08T07:08:58.2484544Z total used free shared buff/cache available 2024-07-08T07:08:58.2485129Z Mem: 6.8Gi 661Mi 4.6Gi 35Mi 1.5Gi 5.8Gi 2024-07-08T07:08:58.2485514Z Swap: 4.0Gi 0B 4.0Gi 2024-07-08T07:08:58.2494128Z Disk Usage: 2024-07-08T07:08:58.2589196Z Filesystem Size Used Avail Use% Mounted on 2024-07-08T07:08:58.2589689Z /dev/root 73G 53G 21G 72% / 2024-07-08T07:08:58.2590174Z tmpfs 3.4G 172K 3.4G 1% /dev/shm 2024-07-08T07:08:58.2590526Z tmpfs 1.4G 1.1M 1.4G 1% /run 2024-07-08T07:08:58.2590886Z tmpfs 5.0M 0 5.0M 0% /run/lock 2024-07-08T07:08:58.2591413Z /dev/sda15 105M 6.1M 99M 6% /boot/efi 2024-07-08T07:08:58.2591816Z /dev/sdb1 14G 4.1G 9.0G 31% /mnt 2024-07-08T07:08:58.2591968Z tmpfs 693M 12K 693M 1% /run/user/1001 2024-07-08T07:08:58.2598390Z Operating System Information: 2024-07-08T07:08:58.2683008Z Linux fv-az458-241 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-07-08T07:08:58.2690952Z Network Configuration: 2024-07-08T07:08:58.2732295Z 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-08T07:08:58.2733180Z link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-08T07:08:58.2733591Z inet 127.0.0.1/8 scope host lo 2024-07-08T07:08:58.2733908Z valid_lft forever preferred_lft forever 2024-07-08T07:08:58.2734246Z inet6 ::1/128 scope host 2024-07-08T07:08:58.2734658Z valid_lft forever preferred_lft forever 2024-07-08T07:08:58.2735352Z 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-08T07:08:58.2735698Z link/ether 00:0d:3a:89:75:75 brd ff:ff:ff:ff:ff:ff 2024-07-08T07:08:58.2735908Z inet 10.1.0.71/16 metric 100 brd 10.1.255.255 scope global eth0 2024-07-08T07:08:58.2736088Z valid_lft forever preferred_lft forever 2024-07-08T07:08:58.2736261Z inet6 fe80::20d:3aff:fe89:7575/64 scope link 2024-07-08T07:08:58.2736440Z valid_lft forever preferred_lft forever 2024-07-08T07:08:58.2736856Z 3: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-07-08T07:08:58.2737087Z link/ether 02:42:7c:9a:2e:5c brd ff:ff:ff:ff:ff:ff 2024-07-08T07:08:58.2737280Z inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-07-08T07:08:58.2737455Z valid_lft forever preferred_lft forever 2024-07-08T07:08:58.2743656Z Docker Version: 2024-07-08T07:08:58.3592015Z Docker version 26.1.3, build b72abbb 2024-07-08T07:08:58.3604853Z Docker Info: 2024-07-08T07:08:58.6579731Z Client: Docker Engine - Community 2024-07-08T07:08:58.6580407Z Version: 26.1.3 2024-07-08T07:08:58.6581028Z Context: default 2024-07-08T07:08:58.6581319Z Debug Mode: false 2024-07-08T07:08:58.6581616Z Plugins: 2024-07-08T07:08:58.6581886Z buildx: Docker Buildx (Docker Inc.) 2024-07-08T07:08:58.6582203Z Version: v0.15.1 2024-07-08T07:08:58.6582563Z Path: /usr/libexec/docker/cli-plugins/docker-buildx 2024-07-08T07:08:58.6728982Z compose: Docker Compose (Docker Inc.) 2024-07-08T07:08:58.6729529Z Version: v2.27.1 2024-07-08T07:08:58.6730438Z Path: /usr/libexec/docker/cli-plugins/docker-compose 2024-07-08T07:08:58.6733636Z 2024-07-08T07:08:58.6733856Z Server: 2024-07-08T07:08:58.6734245Z Containers: 0 2024-07-08T07:08:58.6737388Z Running: 0 2024-07-08T07:08:58.6737604Z Paused: 0 2024-07-08T07:08:58.6737872Z Stopped: 0 2024-07-08T07:08:58.6740984Z Images: 14 2024-07-08T07:08:58.6741206Z Server Version: 26.1.3 2024-07-08T07:08:58.6741489Z Storage Driver: overlay2 2024-07-08T07:08:58.6744614Z Backing Filesystem: extfs 2024-07-08T07:08:58.6744860Z Supports d_type: true 2024-07-08T07:08:58.6745126Z Using metacopy: false 2024-07-08T07:08:58.6748267Z Native Overlay Diff: false 2024-07-08T07:08:58.6748509Z userxattr: false 2024-07-08T07:08:58.6748822Z Logging Driver: json-file 2024-07-08T07:08:58.6752127Z Cgroup Driver: cgroupfs 2024-07-08T07:08:58.6752362Z Cgroup Version: 2 2024-07-08T07:08:58.6752619Z Plugins: 2024-07-08T07:08:58.6755740Z Volume: local 2024-07-08T07:08:58.6755987Z Network: bridge host ipvlan macvlan null overlay 2024-07-08T07:08:58.6756412Z Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog 2024-07-08T07:08:58.6759630Z Swarm: inactive 2024-07-08T07:08:58.6759879Z Runtimes: io.containerd.runc.v2 runc 2024-07-08T07:08:58.6760166Z Default Runtime: runc 2024-07-08T07:08:58.6763318Z Init Binary: docker-init 2024-07-08T07:08:58.6763598Z containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e 2024-07-08T07:08:58.6763966Z runc version: v1.1.13-0-g58aa920 2024-07-08T07:08:58.6767099Z init version: de40ad0 2024-07-08T07:08:58.6767335Z Security Options: 2024-07-08T07:08:58.6767587Z apparmor 2024-07-08T07:08:58.6770700Z seccomp 2024-07-08T07:08:58.6771055Z Profile: builtin 2024-07-08T07:08:58.6771328Z cgroupns 2024-07-08T07:08:58.6774510Z Kernel Version: 6.5.0-1022-azure 2024-07-08T07:08:58.6774763Z Operating System: Ubuntu 22.04.4 LTS 2024-07-08T07:08:58.6775051Z OSType: linux 2024-07-08T07:08:58.6778169Z Architecture: x86_64 2024-07-08T07:08:58.6778405Z CPUs: 2 2024-07-08T07:08:58.6778663Z Total Memory: 6.759GiB 2024-07-08T07:08:58.6781327Z Name: fv-az458-241 2024-07-08T07:08:58.6782505Z ID: 2eba0d88-daa1-420e-94b5-971f98899c95 2024-07-08T07:08:58.6782817Z Docker Root Dir: /var/lib/docker 2024-07-08T07:08:58.6783846Z Debug Mode: false 2024-07-08T07:08:58.6786115Z Username: githubactions 2024-07-08T07:08:58.6787135Z Experimental: false 2024-07-08T07:08:58.6789433Z Insecure Registries: 2024-07-08T07:08:58.6790444Z 127.0.0.0/8 2024-07-08T07:08:58.6792884Z Live Restore Enabled: false 2024-07-08T07:08:58.6793845Z 2024-07-08T07:08:58.7777128Z 2024-07-08T07:08:58.7883755Z ##[section]Finishing: PrintMachineInformation ```

[niteshvijay1995]

@JonathanLydall Thanks for sharing the details. I understand the concern, please do share the details of underlying hardware (PrintMachineInformation output) when the pipeline fails.

[kntajus]

I've just added the task to our pipeline. On the first run, all of the tests failed:

Sample error - 1st run

``` Error Message: Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: ServiceUnavailable (503); Substatus: 20003; ActivityId: a029090b-0318-435f-9755-326ddb47191a; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:07/08/2024 08:19:38; Total Duration:71508.0015 Ms; Request Timeout 65000 Ms; Http Client Timeout:65000 Ms; Activity id: a029090b-0318-435f-9755-326ddb47191a;);; Diagnostics:{"Summary":{"GatewayCalls":{"(0, 0)":3}},"name":"CreateContainerIfNotExistsAsync","start datetime":"2024-07-08T08:19:38.091Z","duration in milliseconds":71509.9027,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T08:18:29.1014963Z","MachineId":"hashedMachineName:286be997-e151-a5b7-ed18-050392d5f074","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.40.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|DN|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:True, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":71509.5562,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":71509.5143,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T08:19:50.5162443Z","cpu":25.649,"memory":3715128.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1321,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:20:00.5168396Z","cpu":21.092,"memory":3715396.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1492,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:20:10.5176154Z","cpu":22.908,"memory":3717592.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.4051,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:20:20.5181921Z","cpu":23.538,"memory":3717640.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1267,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:20:30.5190690Z","cpu":22.261,"memory":3717660.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0648,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:20:40.5196085Z","cpu":22.861,"memory":3725880.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0631,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":71509.4942,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":71509.4862,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":71508.9568,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":71508.9476,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":71508.6911,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T08:19:38.0921123Z","DurationInMs":505.2259,"RequestUri":"[https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a029090b-0318-435f-9755-326ddb47191a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"The](https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container%22,%22ResourceType%22:%22Collection%22,%22HttpMethod%22:%22GET%22,%22ActivityId%22:%22a029090b-0318-435f-9755-326ddb47191a%22,%22ExceptionType%22:%22System.Threading.Tasks.TaskCanceledException%22,%22ExceptionMessage%22:%22The) operation was canceled."},{"StartTimeUTC":"2024-07-08T08:19:38.5974768Z","DurationInMs":5000.1111,"RequestUri":"[https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a029090b-0318-435f-9755-326ddb47191a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"The](https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container%22,%22ResourceType%22:%22Collection%22,%22HttpMethod%22:%22GET%22,%22ActivityId%22:%22a029090b-0318-435f-9755-326ddb47191a%22,%22ExceptionType%22:%22System.Threading.Tasks.TaskCanceledException%22,%22ExceptionMessage%22:%22The) operation was canceled."},{"StartTimeUTC":"2024-07-08T08:19:44.5966845Z","DurationInMs":65002.9137,"RequestUri":"[https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container","ResourceType":"Collection","HttpMethod":"GET","ActivityId":"a029090b-0318-435f-9755-326ddb47191a","ExceptionType":"System.Threading.Tasks.TaskCanceledException","ExceptionMessage":"The](https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container%22,%22ResourceType%22:%22Collection%22,%22HttpMethod%22:%22GET%22,%22ActivityId%22:%22a029090b-0318-435f-9755-326ddb47191a%22,%22ExceptionType%22:%22System.Threading.Tasks.TaskCanceledException%22,%22ExceptionMessage%22:%22The) operation was canceled."}]}}}]}]}]}]}]}]}]} ---- System.Threading.Tasks.TaskCanceledException : The operation was canceled. -------- System.Threading.Tasks.TaskCanceledException : The operation was canceled. ------------ System.IO.IOException : Unable to read data from the transport connection: Operation canceled. ---------------- System.Net.Sockets.SocketException : Operation canceled ```

System Information - 1st run

``` CPU Information: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz CPU family: 6 Model: 85 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: 4 BogoMIPS: 4190.21 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) L2 cache: 2 MiB (2 instances) L3 cache: 35.8 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Vulnerable Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Memory Information: total used free shared buff/cache available Mem: 6.8Gi 1.1Gi 1.4Gi 147Mi 4.3Gi 5.3Gi Swap: 4.0Gi 0B 4.0Gi Disk Usage: Filesystem Size Used Avail Use% Mounted on /dev/root 73G 55G 18G 76% / tmpfs 3.4G 172K 3.4G 1% /dev/shm tmpfs 1.4G 1.1M 1.4G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 14G 4.1G 9.0G 31% /mnt tmpfs 693M 12K 693M 1% /run/user/1001 Operating System Information: Linux fv-az797-993 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Network Configuration: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:22:48:01:91:e3 brd ff:ff:ff:ff:ff:ff inet 10.1.0.11/16 metric 100 brd 10.1.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::222:48ff:fe01:91e3/64 scope link valid_lft forever preferred_lft forever 3: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:fe:e3:26:67 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever Docker Version: Docker version 26.1.3, build b72abbb Docker Info: Client: Docker Engine - Community Version: 26.1.3 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.15.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.27.1 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 14 Server Version: 26.1.3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.5.0-1022-azure Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 6.759GiB Name: fv-az797-993 ID: 2eba0d88-daa1-420e-94b5-971f98899c95 Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ```

I then re-ran the pipeline, and this time only got partial failure, most tests passed but a few didn't:

Sample error - 2nd run

``` Error Message: Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: cc5c2cb8-2c43-400c-adaa-080b7a18df35; Reason: ({"code":"RequestTimeout","message":"Message: Request timed out. More info: [https://aka.ms/cosmosdb-tsg-request-timeout\r\nActivityId](https://aka.ms/cosmosdb-tsg-request-timeout/r/nActivityId): cc5c2cb8-2c43-400c-adaa-080b7a18df35, Request URI: /apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \r\nRequestStartTime: 2024-07-08T08:34:05.5923853Z, RequestEndTime: 2024-07-08T08:34:16.3517294Z, Number of regions attempted:1\r\n{\"systemHistory\":[{\"dateUtc\":\"2024-07-08T08:33:19.0813577Z\",\"cpu\":100.000,\"memory\":3717184.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.2409,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2},{\"dateUtc\":\"2024-07-08T08:33:29.0815904Z\",\"cpu\":100.000,\"memory\":3680828.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":2.0393,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2},{\"dateUtc\":\"2024-07-08T08:33:39.0797900Z\",\"cpu\":100.000,\"memory\":3674388.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.1106,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2},{\"dateUtc\":\"2024-07-08T08:33:49.0833805Z\",\"cpu\":100.000,\"memory\":3673692.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.1597,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2},{\"dateUtc\":\"2024-07-08T08:33:59.0810572Z\",\"cpu\":100.000,\"memory\":3673272.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.0843,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2},{\"dateUtc\":\"2024-07-08T08:34:09.0815964Z\",\"cpu\":100.000,\"memory\":3653784.000,\"threadInfo\":{\"isThreadStarving\":\"False\",\"threadWaitIntervalInMs\":0.1445,\"availableThreads\":32766,\"minThreads\":2,\"maxThreads\":32767},\"numberOfOpenTcpConnection\":2}]}\r\nRequestStart: 2024-07-08T08:34:05.5923853Z; ResponseTime: 2024-07-08T08:34:16.3517294Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.4:10253/apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T08:34:16.3435522Z, activity ID: cc5c2cb8-2c43-400c-adaa-080b7a18df35, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.4:10253/apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.4:43165 -> 172.17.0.4:10253, payload sent: True), BELatencyMs: , ActivityId: cc5c2cb8-2c43-400c-adaa-080b7a18df35, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Unknown | lkt: 07/08/2024 08:33:54)], TransportRequestTimeline: {\"requestTimeline\":[{\"event\": \"Created\", \"startTimeUtc\": \"2024-07-08T08:34:05.5923853Z\", \"durationInMs\": 0.0727},{\"event\": \"ChannelAcquisitionStarted\", \"startTimeUtc\": \"2024-07-08T08:34:05.5924580Z\", \"durationInMs\": 0.0281},{\"event\": \"Pipelined\", \"startTimeUtc\": \"2024-07-08T08:34:05.5924861Z\", \"durationInMs\": 0.1508},{\"event\": \"Transit Time\", \"startTimeUtc\": \"2024-07-08T08:34:05.5926369Z\", \"durationInMs\": 10744.5801},{\"event\": \"Failed\", \"startTimeUtc\": \"2024-07-08T08:34:16.3372170Z\", \"durationInMs\": 0}],\"serviceEndpointStats\":{\"inflightRequests\":1,\"openConnections\":1},\"connectionStats\":{\"waitforConnectionInit\":\"False\",\"callsPendingReceive\":0,\"lastSendAttempt\":\"2024-07-08T08:33:54.3640510Z\",\"lastSend\":\"2024-07-08T08:33:54.3640510Z\",\"lastReceive\":\"2024-07-08T08:33:21.3565348Z\"},\"requestSizeInBytes\":1263,\"requestBodySizeInBytes\":678};\r\n ResourceType: Document, OperationType: Upsert\r\n, SDK: Microsoft.Azure.Documents.Common/2.14.0"} RequestUri: https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container/docs; RequestMethod: POST; Header: Authorization Length: 84; Header: x-ms-date Length: 29; Header: x-ms-documentdb-partitionkey Length: 41; Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1; Header: x-ms-documentdb-is-upsert Length: 4; Header: x-ms-activity-id Length: 36; Header: Cache-Control Length: 8; Header: User-Agent Length: 80; Header: x-ms-version Length: 10; Header: Accept Length: 16; Header: Request-Id Length: 51; Header: traceparent Length: 55; , Request URI: /dbs/external-gateway-database/colls/signature-certificates-container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.34.2);; Diagnostics:{"Summary":{"GatewayCalls":{"(408, 0)":1}},"name":"UpsertItemAsync","start datetime":"2024-07-08T08:34:05.569Z","duration in milliseconds":10798.9133,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T08:33:08.4704734Z","MachineId":"hashedMachineName:a52a5d9e-4a16-e618-7c4c-6929d8e63e4b","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.40.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|DN|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:True, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2},"Synchronization Context":"Xunit.Sdk.AsyncTestSyncContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0587},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":10798.5434,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":10798.4558,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T08:33:20.2899814Z","cpu":87.343,"memory":3692516.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":781.2963,"availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:33:30.2907900Z","cpu":37.620,"memory":3676032.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0403,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:33:40.2912219Z","cpu":16.877,"memory":3674624.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1086,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:33:50.2917839Z","cpu":12.133,"memory":3673692.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0792,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:34:00.2923497Z","cpu":14.365,"memory":3673272.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":788.8634,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T08:34:10.2930501Z","cpu":27.039,"memory":3653784.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.0727,"availableThreads":32765,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":10798.4471,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":10798.439,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":10798.4003,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":10798.3928,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":10797.7812,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T08:34:05.5697679Z","DurationInMs":10795.5655,"RequestUri":"[https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"cc5c2cb8-2c43-400c-adaa-080b7a18df35","StatusCode":"RequestTimeout","ReasonPhrase":"Request](https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container/docs%22,%22ResourceType%22:%22Document%22,%22HttpMethod%22:%22POST%22,%22ActivityId%22:%22cc5c2cb8-2c43-400c-adaa-080b7a18df35%22,%22StatusCode%22:%22RequestTimeout%22,%22ReasonPhrase%22:%22Request) timed out"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"cc5c2cb8-2c43-400c-adaa-080b7a18df35","ResponseTimeUtc":"2024-07-08T08:34:16.3678105Z","StatusCode":408,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/external-gateway-database/colls/signature-certificates-container","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\"code\":\"RequestTimeout\",\"message\":\"Message: Request timed out. More info: [https://aka.ms/cosmosdb-tsg-request-timeout\\r\\nActivityId](https://aka.ms/cosmosdb-tsg-request-timeout//r//nActivityId): cc5c2cb8-2c43-400c-adaa-080b7a18df35, Request URI: /apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\r\\nRequestStartTime: 2024-07-08T08:34:05.5923853Z, RequestEndTime: 2024-07-08T08:34:16.3517294Z, Number of regions attempted:1\\r\\n{\\\"systemHistory\\\":[{\\\"dateUtc\\\":\\\"2024-07-08T08:33:19.0813577Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3717184.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":0.2409,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2},{\\\"dateUtc\\\":\\\"2024-07-08T08:33:29.0815904Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3680828.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":2.0393,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2},{\\\"dateUtc\\\":\\\"2024-07-08T08:33:39.0797900Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3674388.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":0.1106,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2},{\\\"dateUtc\\\":\\\"2024-07-08T08:33:49.0833805Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3673692.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":0.1597,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2},{\\\"dateUtc\\\":\\\"2024-07-08T08:33:59.0810572Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3673272.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":0.0843,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2},{\\\"dateUtc\\\":\\\"2024-07-08T08:34:09.0815964Z\\\",\\\"cpu\\\":100.000,\\\"memory\\\":3653784.000,\\\"threadInfo\\\":{\\\"isThreadStarving\\\":\\\"False\\\",\\\"threadWaitIntervalInMs\\\":0.1445,\\\"availableThreads\\\":32766,\\\"minThreads\\\":2,\\\"maxThreads\\\":32767},\\\"numberOfOpenTcpConnection\\\":2}]}\\r\\nRequestStart: 2024-07-08T08:34:05.5923853Z; ResponseTime: 2024-07-08T08:34:16.3517294Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.4:10253/apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T08:34:16.3435522Z, activity ID: cc5c2cb8-2c43-400c-adaa-080b7a18df35, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.4:10253/apps/DocDbApp/services/DocDbServer9/partitions/a4cb4955-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.4:43165 -> 172.17.0.4:10253, payload sent: True), BELatencyMs: , ActivityId: cc5c2cb8-2c43-400c-adaa-080b7a18df35, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Unknown | lkt: 07/08/2024 08:33:54)], TransportRequestTimeline: {\\\"requestTimeline\\\":[{\\\"event\\\": \\\"Created\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T08:34:05.5923853Z\\\", \\\"durationInMs\\\": 0.0727},{\\\"event\\\": \\\"ChannelAcquisitionStarted\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T08:34:05.5924580Z\\\", \\\"durationInMs\\\": 0.0281},{\\\"event\\\": \\\"Pipelined\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T08:34:05.5924861Z\\\", \\\"durationInMs\\\": 0.1508},{\\\"event\\\": \\\"Transit Time\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T08:34:05.5926369Z\\\", \\\"durationInMs\\\": 10744.5801},{\\\"event\\\": \\\"Failed\\\", \\\"startTimeUtc\\\": \\\"2024-07-08T08:34:16.3372170Z\\\", \\\"durationInMs\\\": 0}],\\\"serviceEndpointStats\\\":{\\\"inflightRequests\\\":1,\\\"openConnections\\\":1},\\\"connectionStats\\\":{\\\"waitforConnectionInit\\\":\\\"False\\\",\\\"callsPendingReceive\\\":0,\\\"lastSendAttempt\\\":\\\"2024-07-08T08:33:54.3640510Z\\\",\\\"lastSend\\\":\\\"2024-07-08T08:33:54.3640510Z\\\",\\\"lastReceive\\\":\\\"2024-07-08T08:33:21.3565348Z\\\"},\\\"requestSizeInBytes\\\":1263,\\\"requestBodySizeInBytes\\\":678};\\r\\n ResourceType: Document, OperationType: Upsert\\r\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\"}\nRequestUri: [https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container/docs;\nRequestMethod](https://127.0.0.1:32769/dbs/external-gateway-database/colls/signature-certificates-container/docs;/nRequestMethod): POST;\nHeader: Authorization Length: 84;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-documentdb-partitionkey Length: 41;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-documentdb-is-upsert Length: 4;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 80;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\nHeader: Request-Id Length: 51;\nHeader: traceparent Length: 55;\n, Request URI: /dbs/external-gateway-database/colls/signature-certificates-container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.34.2\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]}]} Stack Trace: at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request) at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) ```

System Information - 2nd run

``` CPU Information: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz CPU family: 6 Model: 79 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: 1 BogoMIPS: 4589.37 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) L2 cache: 512 KiB (2 instances) L3 cache: 50 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Memory Information: total used free shared buff/cache available Mem: 6.8Gi 1.1Gi 1.4Gi 147Mi 4.3Gi 5.2Gi Swap: 4.0Gi 0B 4.0Gi Disk Usage: Filesystem Size Used Avail Use% Mounted on /dev/root 73G 55G 18G 76% / tmpfs 3.4G 172K 3.4G 1% /dev/shm tmpfs 1.4G 1.1M 1.4G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sdb15 105M 6.1M 99M 6% /boot/efi /dev/sda1 14G 4.1G 9.0G 31% /mnt tmpfs 693M 12K 693M 1% /run/user/1001 Operating System Information: Linux fv-az511-524 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Network Configuration: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:22:48:1b:c7:dc brd ff:ff:ff:ff:ff:ff inet 10.1.28.4/16 metric 100 brd 10.1.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::222:48ff:fe1b:c7dc/64 scope link valid_lft forever preferred_lft forever 3: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:b4:09:ad:c2 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever Docker Version: Docker version 26.1.3, build b72abbb Docker Info: Client: Docker Engine - Community Version: 26.1.3 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.15.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.27.1 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 14 Server Version: 26.1.3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.5.0-1022-azure Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 6.759GiB Name: fv-az511-524 ID: 2eba0d88-daa1-420e-94b5-971f98899c95 Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ```

After another re-run, this time all tests succeeded and we got a green build:

System Information - 3rd run

``` CPU Information: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz CPU family: 6 Model: 79 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: 1 BogoMIPS: 4589.37 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) L2 cache: 512 KiB (2 instances) L3 cache: 50 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Memory Information: total used free shared buff/cache available Mem: 6.8Gi 1.0Gi 1.5Gi 159Mi 4.3Gi 5.3Gi Swap: 4.0Gi 0B 4.0Gi Disk Usage: Filesystem Size Used Avail Use% Mounted on /dev/root 73G 55G 18G 76% / tmpfs 3.4G 172K 3.4G 1% /dev/shm tmpfs 1.4G 1.1M 1.4G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda15 105M 6.1M 99M 6% /boot/efi /dev/sdb1 14G 4.1G 9.0G 31% /mnt tmpfs 693M 12K 693M 1% /run/user/1001 Operating System Information: Linux fv-az511-524 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Network Configuration: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:22:48:c7:5f:bc brd ff:ff:ff:ff:ff:ff inet 10.1.178.2/16 metric 100 brd 10.1.255.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::222:48ff:fec7:5fbc/64 scope link valid_lft forever preferred_lft forever 3: docker0: mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:6c:0c:0d:82 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever Docker Version: Docker version 26.1.3, build b72abbb Docker Info: Client: Docker Engine - Community Version: 26.1.3 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.15.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.27.1 Path: /usr/libexec/docker/cli-plugins/docker-compose Server: Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 14 Server Version: 26.1.3 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: false userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog Swarm: inactive Runtimes: runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e runc version: v1.1.13-0-g58aa920 init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.5.0-1022-azure Operating System: Ubuntu 22.04.4 LTS OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 6.759GiB Name: fv-az511-524 ID: 2eba0d88-daa1-420e-94b5-971f98899c95 Docker Root Dir: /var/lib/docker Debug Mode: false Username: githubactions Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ```

Hope that helps.

[niteshvijay1995]

I've just added the task to our pipeline. On the first run, all of the tests failed:

Sample error - 1st run System Information - 1st run

@kntajus, Please attach the yaml file. Also, can you please verify if you observe similar test failure when running it on any local machine?

[kntajus]

@niteshvijay1995 What is it that you're interested in within the YAML? I ask because I'm not sure if there's anything sensitive in there that I shouldn't be sharing and it may take me a while to work that out.

In terms of the tests being run, they are C# XUnit tests that are using TestContainers to spin up the docker container, and the YAML for that task is simply:

          - task: DotNetCoreCLI@2
            displayName: Run Tests
            inputs:
              command: test
              projects: |
                tests/**/*.IntegrationTests.csproj
              publishTestResults: true
              arguments: '--no-build -c $(build.configuration) /p:Platform=x64 --collect:"XPlat Code Coverage"'

I can confirm that we've had 100% success on local runs for many months (we introduced this near the start of this year), it's only ever failed (intermittently) when running in Azure DevOps pipelines.

[JonathanLydall]

Hi @niteshvijay1995,

We had a failure yesterday (in addition to a few more builds which were fine).

Machine information for failed build

``` 2024-07-08T10:08:36.1616463Z ##[section]Starting: PrintMachineInformation 2024-07-08T10:08:36.1860522Z ============================================================================== 2024-07-08T10:08:36.1861215Z Task : PowerShell 2024-07-08T10:08:36.1861458Z Description : Run a PowerShell script on Linux, macOS, or Windows 2024-07-08T10:08:36.1861577Z Version : 2.239.1 2024-07-08T10:08:36.1861779Z Author : Microsoft Corporation 2024-07-08T10:08:36.1862096Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell 2024-07-08T10:08:36.1862221Z ============================================================================== 2024-07-08T10:08:36.5719631Z Generating script. 2024-07-08T10:08:36.5768969Z ========================== Starting Command Output =========================== 2024-07-08T10:08:36.5901685Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/55bc2847-d486-47c6-ae07-e0e5a259cbe2.ps1' 2024-07-08T10:08:37.8379995Z CPU Information: 2024-07-08T10:08:37.8592842Z Architecture: x86_64 2024-07-08T10:08:37.8593319Z CPU op-mode(s): 32-bit, 64-bit 2024-07-08T10:08:37.8593518Z Address sizes: 46 bits physical, 48 bits virtual 2024-07-08T10:08:37.8593691Z Byte Order: Little Endian 2024-07-08T10:08:37.8593847Z CPU(s): 2 2024-07-08T10:08:37.8594038Z On-line CPU(s) list: 0,1 2024-07-08T10:08:37.8594207Z Vendor ID: GenuineIntel 2024-07-08T10:08:37.8594386Z Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz 2024-07-08T10:08:37.8594575Z CPU family: 6 2024-07-08T10:08:37.8594723Z Model: 85 2024-07-08T10:08:37.8594857Z Thread(s) per core: 1 2024-07-08T10:08:37.8595027Z Core(s) per socket: 2 2024-07-08T10:08:37.8595161Z Socket(s): 1 2024-07-08T10:08:37.8595307Z Stepping: 7 2024-07-08T10:08:37.8595453Z BogoMIPS: 5187.81 2024-07-08T10:08:37.8595969Z Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear 2024-07-08T10:08:37.8596496Z Hypervisor vendor: Microsoft 2024-07-08T10:08:37.8596664Z Virtualization type: full 2024-07-08T10:08:37.8596821Z L1d cache: 64 KiB (2 instances) 2024-07-08T10:08:37.8596999Z L1i cache: 64 KiB (2 instances) 2024-07-08T10:08:37.8597156Z L2 cache: 2 MiB (2 instances) 2024-07-08T10:08:37.8597330Z L3 cache: 35.8 MiB (1 instance) 2024-07-08T10:08:37.8597491Z NUMA node(s): 1 2024-07-08T10:08:37.8597631Z NUMA node0 CPU(s): 0,1 2024-07-08T10:08:37.8597835Z Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status 2024-07-08T10:08:37.8598063Z Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported 2024-07-08T10:08:37.8598270Z Vulnerability L1tf: Mitigation; PTE Inversion 2024-07-08T10:08:37.8598471Z Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T10:08:37.8598713Z Vulnerability Meltdown: Mitigation; PTI 2024-07-08T10:08:37.8598953Z Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown 2024-07-08T10:08:37.8599465Z Vulnerability Retbleed: Vulnerable 2024-07-08T10:08:37.8599656Z Vulnerability Spec rstack overflow: Not affected 2024-07-08T10:08:37.8599982Z Vulnerability Spec store bypass: Vulnerable 2024-07-08T10:08:37.8600217Z Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization 2024-07-08T10:08:37.8600675Z Vulnerability Spectre v2: Mitigation; Retpolines; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Retpoline 2024-07-08T10:08:37.8600907Z Vulnerability Srbds: Not affected 2024-07-08T10:08:37.8601120Z Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown 2024-07-08T10:08:37.8605504Z Memory Information: 2024-07-08T10:08:37.8702673Z total used free shared buff/cache available 2024-07-08T10:08:37.8703286Z Mem: 6.8Gi 632Mi 4.7Gi 35Mi 1.5Gi 5.8Gi 2024-07-08T10:08:37.8703996Z Swap: 4.0Gi 0B 4.0Gi 2024-07-08T10:08:37.8705513Z Disk Usage: 2024-07-08T10:08:37.8847315Z Filesystem Size Used Avail Use% Mounted on 2024-07-08T10:08:37.8847530Z /dev/root 73G 53G 21G 72% / 2024-07-08T10:08:37.8847701Z tmpfs 3.4G 172K 3.4G 1% /dev/shm 2024-07-08T10:08:37.8847856Z tmpfs 1.4G 1.1M 1.4G 1% /run 2024-07-08T10:08:37.8848027Z tmpfs 5.0M 0 5.0M 0% /run/lock 2024-07-08T10:08:37.8848199Z /dev/sdb15 105M 6.1M 99M 6% /boot/efi 2024-07-08T10:08:37.8848361Z /dev/sda1 14G 4.1G 9.0G 31% /mnt 2024-07-08T10:08:37.8848541Z tmpfs 693M 12K 693M 1% /run/user/1001 2024-07-08T10:08:37.8848700Z Operating System Information: 2024-07-08T10:08:37.8904271Z Linux fv-az356-146 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-07-08T10:08:37.8910564Z Network Configuration: 2024-07-08T10:08:37.8956344Z 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-07-08T10:08:37.8957053Z link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-07-08T10:08:37.8959072Z inet 127.0.0.1/8 scope host lo 2024-07-08T10:08:37.8959511Z valid_lft forever preferred_lft forever 2024-07-08T10:08:37.8960267Z inet6 ::1/128 scope host 2024-07-08T10:08:37.8960616Z valid_lft forever preferred_lft forever 2024-07-08T10:08:37.8961379Z 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-07-08T10:08:37.8961805Z link/ether 00:0d:3a:da:86:c4 brd ff:ff:ff:ff:ff:ff 2024-07-08T10:08:37.8962567Z inet 10.1.28.0/16 metric 100 brd 10.1.255.255 scope global eth0 2024-07-08T10:08:37.8962941Z valid_lft forever preferred_lft forever 2024-07-08T10:08:37.8963633Z inet6 fe80::20d:3aff:feda:86c4/64 scope link 2024-07-08T10:08:37.8964182Z valid_lft forever preferred_lft forever 2024-07-08T10:08:37.8965260Z 3: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-07-08T10:08:37.8965714Z link/ether 02:42:35:c8:95:4a brd ff:ff:ff:ff:ff:ff 2024-07-08T10:08:37.8966880Z inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-07-08T10:08:37.8967262Z valid_lft forever preferred_lft forever 2024-07-08T10:08:37.8968037Z Docker Version: 2024-07-08T10:08:37.9780914Z Docker version 26.1.3, build b72abbb 2024-07-08T10:08:37.9793433Z Docker Info: 2024-07-08T10:08:38.2569774Z Client: Docker Engine - Community 2024-07-08T10:08:38.2571132Z Version: 26.1.3 2024-07-08T10:08:38.2574187Z Context: default 2024-07-08T10:08:38.2574550Z Debug Mode: false 2024-07-08T10:08:38.2574691Z Plugins: 2024-07-08T10:08:38.2574883Z buildx: Docker Buildx (Docker Inc.) 2024-07-08T10:08:38.2575046Z Version: v0.15.1 2024-07-08T10:08:38.2575404Z Path: /usr/libexec/docker/cli-plugins/docker-buildx 2024-07-08T10:08:38.2575599Z compose: Docker Compose (Docker Inc.) 2024-07-08T10:08:38.2575744Z Version: v2.27.1 2024-07-08T10:08:38.2576229Z Path: /usr/libexec/docker/cli-plugins/docker-compose 2024-07-08T10:08:38.2576318Z 2024-07-08T10:08:38.2576438Z Server: 2024-07-08T10:08:38.2576660Z Containers: 0 2024-07-08T10:08:38.2576796Z Running: 0 2024-07-08T10:08:38.2576928Z Paused: 0 2024-07-08T10:08:38.2577059Z Stopped: 0 2024-07-08T10:08:38.2577174Z Images: 14 2024-07-08T10:08:38.2577322Z Server Version: 26.1.3 2024-07-08T10:08:38.2577478Z Storage Driver: overlay2 2024-07-08T10:08:38.2577621Z Backing Filesystem: extfs 2024-07-08T10:08:38.2577778Z Supports d_type: true 2024-07-08T10:08:38.2577916Z Using metacopy: false 2024-07-08T10:08:38.2578070Z Native Overlay Diff: false 2024-07-08T10:08:38.2578209Z userxattr: false 2024-07-08T10:08:38.2578385Z Logging Driver: json-file 2024-07-08T10:08:38.2578526Z Cgroup Driver: cgroupfs 2024-07-08T10:08:38.2578676Z Cgroup Version: 2 2024-07-08T10:08:38.2578810Z Plugins: 2024-07-08T10:08:38.2578929Z Volume: local 2024-07-08T10:08:38.2579271Z Network: bridge host ipvlan macvlan null overlay 2024-07-08T10:08:38.2579736Z Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog 2024-07-08T10:08:38.2580020Z Swarm: inactive 2024-07-08T10:08:38.2580262Z Runtimes: io.containerd.runc.v2 runc 2024-07-08T10:08:38.2580497Z Default Runtime: runc 2024-07-08T10:08:38.2580753Z Init Binary: docker-init 2024-07-08T10:08:38.2581014Z containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e 2024-07-08T10:08:38.2581352Z runc version: v1.1.13-0-g58aa920 2024-07-08T10:08:38.2581596Z init version: de40ad0 2024-07-08T10:08:38.2581809Z Security Options: 2024-07-08T10:08:38.2582024Z apparmor 2024-07-08T10:08:38.2582217Z seccomp 2024-07-08T10:08:38.2582450Z Profile: builtin 2024-07-08T10:08:38.2582664Z cgroupns 2024-07-08T10:08:38.2582931Z Kernel Version: 6.5.0-1022-azure 2024-07-08T10:08:38.2583189Z Operating System: Ubuntu 22.04.4 LTS 2024-07-08T10:08:38.2583411Z OSType: linux 2024-07-08T10:08:38.2583637Z Architecture: x86_64 2024-07-08T10:08:38.2583856Z CPUs: 2 2024-07-08T10:08:38.2584058Z Total Memory: 6.759GiB 2024-07-08T10:08:38.2584327Z Name: fv-az356-146 2024-07-08T10:08:38.2584633Z ID: 2eba0d88-daa1-420e-94b5-971f98899c95 2024-07-08T10:08:38.2584892Z Docker Root Dir: /var/lib/docker 2024-07-08T10:08:38.2585121Z Debug Mode: false 2024-07-08T10:08:38.2585340Z Username: githubactions 2024-07-08T10:08:38.2585572Z Experimental: false 2024-07-08T10:08:38.2585799Z Insecure Registries: 2024-07-08T10:08:38.2586020Z 127.0.0.0/8 2024-07-08T10:08:38.2586247Z Live Restore Enabled: false 2024-07-08T10:08:38.2586391Z 2024-07-08T10:08:38.3318420Z 2024-07-08T10:08:38.3427956Z ##[section]Finishing: PrintMachineInformation ```

Example of error of failing test

``` 'Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: fdc46b20-f040-4dd1-8557-e5def5567113; Reason: ({"code":"RequestTimeout","message":"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\r\\nActivityId: fdc46b20-f040-4dd1-8557-e5def5567113, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\r\\nRequestStartTime: 2024-07-08T09:23:35.6622449Z, RequestEndTime: 2024-07-08T09:23:46.3008637Z, Number of regions attempted:1\\r\\n{\\"systemHistory\\":[{\\"dateUtc\\":\\"2024-07-08T09:23:28.1280875Z\\",\\"cpu\\":100.000,\\"memory\\":3343544.000,\\"threadInfo\\":{\\"isThreadStarving\\":\\"no info\\",\\"availableThreads\\":32765,\\"minThreads\\":2,\\"maxThreads\\":32767},\\"numberOfOpenTcpConnection\\":0},{\\"dateUtc\\":\\"2024-07-08T09:23:38.1485238Z\\",\\"cpu\\":100.000,\\"memory\\":3332932.000,\\"threadInfo\\":{\\"isThreadStarving\\":\\"False\\",\\"threadWaitIntervalInMs\\":0.2478,\\"availableThreads\\":32766,\\"minThreads\\":2,\\"maxThreads\\":32767},\\"numberOfOpenTcpConnection\\":2}]}\\r\\nRequestStart: 2024-07-08T09:23:35.6622449Z; ResponseTime: 2024-07-08T09:23:46.3008637Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T09:23:46.2927051Z, activity ID: fdc46b20-f040-4dd1-8557-e5def5567113, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:52101 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: fdc46b20-f040-4dd1-8557-e5def5567113, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 09:23:31)], TransportRequestTimeline: {\\"requestTimeline\\":[{\\"event\\": \\"Created\\", \\"startTimeUtc\\": \\"2024-07-08T09:23:35.6622449Z\\", \\"durationInMs\\": 0.0845},{\\"event\\": \\"ChannelAcquisitionStarted\\", \\"startTimeUtc\\": \\"2024-07-08T09:23:35.6623294Z\\", \\"durationInMs\\": 0.0025},{\\"event\\": \\"Pipelined\\", \\"startTimeUtc\\": \\"2024-07-08T09:23:35.6623319Z\\", \\"durationInMs\\": 0.6984},{\\"event\\": \\"Transit Time\\", \\"startTimeUtc\\": \\"2024-07-08T09:23:35.6630303Z\\", \\"durationInMs\\": 10630.2103},{\\"event\\": \\"Failed\\", \\"startTimeUtc\\": \\"2024-07-08T09:23:46.2932406Z\\", \\"durationInMs\\": 0}],\\"serviceEndpointStats\\":{\\"inflightRequests\\":1,\\"openConnections\\":1},\\"connectionStats\\":{\\"waitforConnectionInit\\":\\"False\\",\\"callsPendingReceive\\":0,\\"lastSendAttempt\\":\\"2024-07-08T09:23:35.6265947Z\\",\\"lastSend\\":\\"2024-07-08T09:23:35.6265947Z\\",\\"lastReceive\\":\\"2024-07-08T09:23:35.6265947Z\\"},\\"requestSizeInBytes\\":1001,\\"requestBodySizeInBytes\\":446};\\r\\n ResourceType: Document, OperationType: Create\\r\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0"}\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\nRequestMethod: POST;\nHeader: Authorization Length: 84;\nHeader: x-ms-date Length: 29;\nHeader: x-ms-documentdb-partitionkey Length: 40;\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\nHeader: x-ms-activity-id Length: 36;\nHeader: Cache-Control Length: 8;\nHeader: User-Agent Length: 79;\nHeader: x-ms-version Length: 10;\nHeader: Accept Length: 16;\nHeader: traceparent Length: 55;\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0);\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)\n--- Cosmos Diagnostics ---{"Summary":{"GatewayCalls":{"(408, 0)":1}},"name":"CreateItemAsync","start datetime":"2024-07-08T09:23:35.661Z","duration in milliseconds":10665.9224,"data":{"Client Configuration":{"Client Created Time Utc":"2024-07-08T09:23:26.2923431Z","MachineId":"hashedMachineName:b99417d1-80fa-f678-0803-6b772118e5e0","NumberOfClientsCreated":1,"NumberOfActiveClients":1,"ConnectionMode":"Gateway","User Agent":"cosmos-netstandard-sdk/3.38.0|1|X64|Ubuntu 22.04.4 LTS|.NET 8.0.6|N|F 00000010|","ConnectionConfig":{"gw":"(cps:50, urto:6, p:False, httpf: True)","rntbd":"(cto: 5, icto: -1, mrpc: 30, mcpe: 65535, erd: True, pr: ReuseUnicastPort)","other":"(ed:False, be:False)"},"ConsistencyConfig":"(consistency: NotSet, prgns:[], apprgn: )","ProcessorCount":2}},"children":[{"name":"ItemSerialize","duration in milliseconds":0.0633},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":10665.6232,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":10665.5877,"data":{"System Info":{"systemHistory":[{"dateUtc":"2024-07-08T09:23:29.1845225Z","cpu":0.000,"memory":3321776.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32763,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0},{"dateUtc":"2024-07-08T09:23:39.1947613Z","cpu":59.759,"memory":3332956.000,"threadInfo":{"isThreadStarving":"False","threadWaitIntervalInMs":0.1821,"availableThreads":32764,"minThreads":2,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TelemetryHandler","duration in milliseconds":10665.5774,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":10665.5665,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":10665.4393,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":10665.4343,"children":[{"name":"Microsoft.Azure.Cosmos.GatewayStoreModel Transport Request","duration in milliseconds":10660.4866,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[],"HttpResponseStats":[{"StartTimeUTC":"2024-07-08T09:23:35.6614139Z","DurationInMs":10643.2811,"RequestUri":"https://localhost:32769/dbs/TestDb/colls/Container/docs","ResourceType":"Document","HttpMethod":"POST","ActivityId":"fdc46b20-f040-4dd1-8557-e5def5567113","StatusCode":"RequestTimeout","ReasonPhrase":"Request timed out"}]},"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"fdc46b20-f040-4dd1-8557-e5def5567113","ResponseTimeUtc":"2024-07-08T09:23:46.3245759Z","StatusCode":408,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/TestDb/colls/Container","ErrorMessage":"Microsoft.Azure.Documents.DocumentClientException: {\\"code\\":\\"RequestTimeout\\",\\"message\\":\\"Message: Request timed out. More info: https://aka.ms/cosmosdb-tsg-request-timeout\\\\r\\\\nActivityId: fdc46b20-f040-4dd1-8557-e5def5567113, Request URI: /apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, RequestStats: \\\\r\\\\nRequestStartTime: 2024-07-08T09:23:35.6622449Z, RequestEndTime: 2024-07-08T09:23:46.3008637Z, Number of regions attempted:1\\\\r\\\\n{\\\\\\"systemHistory\\\\\\":[{\\\\\\"dateUtc\\\\\\":\\\\\\"2024-07-08T09:23:28.1280875Z\\\\\\",\\\\\\"cpu\\\\\\":100.000,\\\\\\"memory\\\\\\":3343544.000,\\\\\\"threadInfo\\\\\\":{\\\\\\"isThreadStarving\\\\\\":\\\\\\"no info\\\\\\",\\\\\\"availableThreads\\\\\\":32765,\\\\\\"minThreads\\\\\\":2,\\\\\\"maxThreads\\\\\\":32767},\\\\\\"numberOfOpenTcpConnection\\\\\\":0},{\\\\\\"dateUtc\\\\\\":\\\\\\"2024-07-08T09:23:38.1485238Z\\\\\\",\\\\\\"cpu\\\\\\":100.000,\\\\\\"memory\\\\\\":3332932.000,\\\\\\"threadInfo\\\\\\":{\\\\\\"isThreadStarving\\\\\\":\\\\\\"False\\\\\\",\\\\\\"threadWaitIntervalInMs\\\\\\":0.2478,\\\\\\"availableThreads\\\\\\":32766,\\\\\\"minThreads\\\\\\":2,\\\\\\"maxThreads\\\\\\":32767},\\\\\\"numberOfOpenTcpConnection\\\\\\":2}]}\\\\r\\\\nRequestStart: 2024-07-08T09:23:35.6622449Z; ResponseTime: 2024-07-08T09:23:46.3008637Z; StoreResult: StorePhysicalAddress: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, LSN: -1, GlobalCommittedLsn: -1, PartitionKeyRangeId: , IsValid: False, StatusCode: 408, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: , UsingLocalLSN: False, TransportException: A client transport error occurred: The request timed out while waiting for a server response. (Time: 2024-07-08T09:23:46.2927051Z, activity ID: fdc46b20-f040-4dd1-8557-e5def5567113, error code: ReceiveTimeout [0x0010], base error: HRESULT 0x80131500, URI: rntbd://172.17.0.3:10253/apps/DocDbApp/services/DocDbServer1/partitions/a4cb494d-38c8-11e6-8106-8cdcd42c33be/replicas/1p/, connection: 172.17.0.3:52101 -> 172.17.0.3:10253, payload sent: True), BELatencyMs: , ActivityId: fdc46b20-f040-4dd1-8557-e5def5567113, RetryAfterInMs: , ReplicaHealthStatuses: [(port: 10253 | status: Connected | lkt: 07/08/2024 09:23:31)], TransportRequestTimeline: {\\\\\\"requestTimeline\\\\\\":[{\\\\\\"event\\\\\\": \\\\\\"Created\\\\\\", \\\\\\"startTimeUtc\\\\\\": \\\\\\"2024-07-08T09:23:35.6622449Z\\\\\\", \\\\\\"durationInMs\\\\\\": 0.0845},{\\\\\\"event\\\\\\": \\\\\\"ChannelAcquisitionStarted\\\\\\", \\\\\\"startTimeUtc\\\\\\": \\\\\\"2024-07-08T09:23:35.6623294Z\\\\\\", \\\\\\"durationInMs\\\\\\": 0.0025},{\\\\\\"event\\\\\\": \\\\\\"Pipelined\\\\\\", \\\\\\"startTimeUtc\\\\\\": \\\\\\"2024-07-08T09:23:35.6623319Z\\\\\\", \\\\\\"durationInMs\\\\\\": 0.6984},{\\\\\\"event\\\\\\": \\\\\\"Transit Time\\\\\\", \\\\\\"startTimeUtc\\\\\\": \\\\\\"2024-07-08T09:23:35.6630303Z\\\\\\", \\\\\\"durationInMs\\\\\\": 10630.2103},{\\\\\\"event\\\\\\": \\\\\\"Failed\\\\\\", \\\\\\"startTimeUtc\\\\\\": \\\\\\"2024-07-08T09:23:46.2932406Z\\\\\\", \\\\\\"durationInMs\\\\\\": 0}],\\\\\\"serviceEndpointStats\\\\\\":{\\\\\\"inflightRequests\\\\\\":1,\\\\\\"openConnections\\\\\\":1},\\\\\\"connectionStats\\\\\\":{\\\\\\"waitforConnectionInit\\\\\\":\\\\\\"False\\\\\\",\\\\\\"callsPendingReceive\\\\\\":0,\\\\\\"lastSendAttempt\\\\\\":\\\\\\"2024-07-08T09:23:35.6265947Z\\\\\\",\\\\\\"lastSend\\\\\\":\\\\\\"2024-07-08T09:23:35.6265947Z\\\\\\",\\\\\\"lastReceive\\\\\\":\\\\\\"2024-07-08T09:23:35.6265947Z\\\\\\"},\\\\\\"requestSizeInBytes\\\\\\":1001,\\\\\\"requestBodySizeInBytes\\\\\\":446};\\\\r\\\\n ResourceType: Document, OperationType: Create\\\\r\\\\n, SDK: Microsoft.Azure.Documents.Common/2.14.0\\"}\\nRequestUri: https://localhost:32769/dbs/TestDb/colls/Container/docs;\\nRequestMethod: POST;\\nHeader: Authorization Length: 84;\\nHeader: x-ms-date Length: 29;\\nHeader: x-ms-documentdb-partitionkey Length: 40;\\nHeader: x-ms-cosmos-sdk-supportedcapabilities Length: 1;\\nHeader: x-ms-activity-id Length: 36;\\nHeader: Cache-Control Length: 8;\\nHeader: User-Agent Length: 79;\\nHeader: x-ms-version Length: 10;\\nHeader: Accept Length: 16;\\nHeader: traceparent Length: 55;\\n, Request URI: /dbs/TestDb/colls/Container/docs, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Linux/22.04 cosmos-netstandard-sdk/3.32.0\\n at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request)\\n at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":null,"BELatencyInMs":null}}}]}]}]}]}]}]}]}' ```

Diff of the machine information

```diff diff --git "a/machine-info-success.txt" "b/machine-info-failure.txt" index 6379f46..9ff21b4 100644 --- "a/machine-info-success.txt" +++ "b/machine-info-failure.txt" @@ -8,7 +8,7 @@ Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/p ============================================================================== Generating script. ========================== Starting Command Output =========================== -[command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/ec24e82d-e7fa-4de5-95c5-306e9ec7ba53.ps1' +[command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -Command . '/home/vsts/work/_temp/55bc2847-d486-47c6-ae07-e0e5a259cbe2.ps1' CPU Information: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit @@ -17,30 +17,30 @@ Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Vendor ID: GenuineIntel -Model name: Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz +Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz CPU family: 6 -Model: 79 +Model: 85 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 -Stepping: 1 -BogoMIPS: 4589.37 -Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt md_clear +Stepping: 7 +BogoMIPS: 5187.81 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) -L2 cache: 512 KiB (2 instances) -L3 cache: 50 MiB (1 instance) +L2 cache: 2 MiB (2 instances) +L3 cache: 35.8 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0,1 -Vulnerability Gather data sampling: Not affected +Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported Vulnerability L1tf: Mitigation; PTE Inversion Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown Vulnerability Meltdown: Mitigation; PTI Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown -Vulnerability Retbleed: Not affected +Vulnerability Retbleed: Vulnerable Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Vulnerable Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization @@ -49,11 +49,11 @@ Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown Memory Information: total used free shared buff/cache available -Mem: 6.8Gi 672Mi 4.6Gi 35Mi 1.5Gi 5.8Gi +Mem: 6.8Gi 632Mi 4.7Gi 35Mi 1.5Gi 5.8Gi Swap: 4.0Gi 0B 4.0Gi Disk Usage: Filesystem Size Used Avail Use% Mounted on -/dev/root 73G 52G 21G 72% / +/dev/root 73G 53G 21G 72% / tmpfs 3.4G 172K 3.4G 1% /dev/shm tmpfs 1.4G 1.1M 1.4G 1% /run tmpfs 5.0M 0 5.0M 0% /run/lock @@ -61,7 +61,7 @@ tmpfs 5.0M 0 5.0M 0% /run/lock /dev/sda1 14G 4.1G 9.0G 31% /mnt tmpfs 693M 12K 693M 1% /run/user/1001 Operating System Information: -Linux fv-az367-389 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux +Linux fv-az356-146 6.5.0-1022-azure #23~22.04.1-Ubuntu SMP Thu May 9 17:59:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Network Configuration: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 @@ -70,13 +70,13 @@ Network Configuration: inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1500 qdisc mq state UP group default qlen 1000 - link/ether 00:22:48:9a:99:5f brd ff:ff:ff:ff:ff:ff - inet 10.1.0.5/16 metric 100 brd 10.1.255.255 scope global eth0 + link/ether 00:0d:3a:da:86:c4 brd ff:ff:ff:ff:ff:ff + inet 10.1.28.0/16 metric 100 brd 10.1.255.255 scope global eth0 valid_lft forever preferred_lft forever - inet6 fe80::222:48ff:fe9a:995f/64 scope link + inet6 fe80::20d:3aff:feda:86c4/64 scope link valid_lft forever preferred_lft forever 3: docker0: mtu 1500 qdisc noqueue state DOWN group default - link/ether 02:42:0e:68:7c:0b brd ff:ff:ff:ff:ff:ff + link/ether 02:42:35:c8:95:4a brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever Docker Version: @@ -132,7 +132,7 @@ Server: Architecture: x86_64 CPUs: 2 Total Memory: 6.759GiB - Name: fv-az367-389 + Name: fv-az356-146 ID: 2eba0d88-daa1-420e-94b5-971f98899c95 Docker Root Dir: /var/lib/docker Debug Mode: false ```