Retry loop in ContainerReconciler exhausts system RAM

klazuka commented 1 day ago

Is there an existing issue for this?

[x] I have searched the existing issues

Describe the bug

When the container runtime is not healthy, Aspire retries forever with no delay, eventually exhausting system memory.

We recently upgraded to Aspire 9.0 rc1 and started using its ability to manage a Docker container to run Azurite. This morning I tried to run my .net app, but I forgot to pass the run flag that would also start the Azurite container. During the boot process, Aspire continually retried waiting for the container to be healthy, spewing logs to stdout so fast that eventually my Terminal died once it exhausted system memory. The process was using 29GB when I terminated it, nearly all of that consisting of retry log messages (see screenshot)

Expected Behavior

When retrying, it should backoff and/or eventually give up. It should not run a hot loop with no delay.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version info

> dotnet --info
.NET SDK:
 Version:           9.0.100-rc.2.24474.11
 Commit:            315e1305db
 Workload version:  9.0.100-manifests.cea95dba
 MSBuild version:   17.12.0-preview-24473-03+fea15fbd1

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  15.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/9.0.100-rc.2.24474.11/

.NET workloads installed:
There are no installed workloads to display.
Configured to use loose manifests when installing new manifests.

Host:
  Version:      9.0.0-rc.2.24473.5
  Architecture: arm64
  Commit:       990ebf52fc

.NET SDKs installed:
  8.0.304 [/usr/local/share/dotnet/sdk]
  9.0.100-rc.2.24474.11 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 9.0.0-rc.2.24474.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.0-rc.2.24473.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

Anything else?

Aspire 9.0 RC1 9.0.0-rc.1.24511.1

MarcinJuraszek commented 1 day ago

cc: @davidfowl You asked us to try the containerized version of Azurite, so we did, and so far it's working ok, except for the above :)

joperezr commented 20 hours ago

Thanks for logging this @MarcinJuraszek, we believe this is a dupe of https://github.com/dotnet/aspire/issues/6341 and was fixed in 9.0. Can you try the latest build of 9.1 (builds from main) to see if you can repro?

MarcinJuraszek commented 17 hours ago

How would one go about trying "builds from main"? Do you have any instructions you could share?

corrosive4354 commented 17 hours ago

This is an obvious bug I see you are using macos, you can try servbay, which is better than docker

dotnet / aspire