search

dotnet / aspire

Tools, templates, and packages to accelerate building observable, production-ready apps
https://learn.microsoft.com/dotnet/aspire
MIT License
3.94k stars 483 forks source link

Aspire Postgres and PGAdmin healthchecks failing #6818

Open cisionmarkwalls opened 3 days ago

cisionmarkwalls commented 3 days ago

Is there an existing issue for this?

Describe the bug

I upgraded from Aspire 8 to 9. The dashboard itself is running correctly. I have pulled the PG and PGAdmin docker images that are the defaults for 9.

I added postgres and pgadmin like this:

var postgres = builder.AddPostgres("DefaultConnection").WithPgAdmin();

This is the error that the Postgres image gives in the Aspire dashboard:

System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.<SendCoreAsync>g__Core|4_0(HttpRequestMessage request, Boolean useAsync, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at HealthChecks.Uris.UriHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken) in /_/src/HealthChecks.Uris/UriHealthCheck.cs:line 54

Looking into the container details for the postgres image from VS I get:

{
  "Id": "f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a",
  "Created": "2024-11-27T01:59:49.22771279Z",
  "Path": "docker-entrypoint.sh",
  "Args": [
    "postgres"
  ],
  "State": {
    "Status": "running",
    "Running": true,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 19224,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "2024-11-27T01:59:51.160971086Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
  },
  "Image": "sha256:2bb72c2c69f9f837d0e2a831b8e4772869a933dd8e81f4ddf19af1c97ab5c7fb",
  "ResolvConfPath": "/var/lib/docker/containers/f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a/resolv.conf",
  "HostnamePath": "/var/lib/docker/containers/f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a/hostname",
  "HostsPath": "/var/lib/docker/containers/f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a/hosts",
  "LogPath": "/var/lib/docker/containers/f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a/f904f182413982863d4a5a7a9c210c278ed5a6f7217c411a475ba0e3454c4d6a-json.log",
  "Name": "/DefaultConnection-svfstknz",
  "RestartCount": 0,
  "Driver": "overlay2",
  "Platform": "linux",
  "MountLabel": "",
  "ProcessLabel": "",
  "AppArmorProfile": "",
  "ExecIDs": null,
  "HostConfig": {
    "Binds": null,
    "ContainerIDFile": "",
    "LogConfig": {
      "Type": "json-file",
      "Config": {}
    },
    "NetworkMode": "bridge",
    "PortBindings": {
      "5432/tcp": [
        {
          "HostIp": "127.0.0.1",
          "HostPort": ""
        }
      ]
    },
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": null,
    "ConsoleSize": [
      0,
      0
    ],
    "CapAdd": null,
    "CapDrop": null,
    "CgroupnsMode": "host",
    "Dns": [],
    "DnsOptions": [],
    "DnsSearch": [],
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "private",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": false,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": null,
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 67108864,
    "Runtime": "runc",
    "Isolation": "",
    "CpuShares": 0,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "",
    "BlkioWeight": 0,
    "BlkioWeightDevice": [],
    "BlkioDeviceReadBps": [],
    "BlkioDeviceWriteBps": [],
    "BlkioDeviceReadIOps": [],
    "BlkioDeviceWriteIOps": [],
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": [],
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": null,
    "OomKillDisable": false,
    "PidsLimit": null,
    "Ulimits": [],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": [
      "/proc/asound",
      "/proc/acpi",
      "/proc/kcore",
      "/proc/keys",
      "/proc/latency_stats",
      "/proc/timer_list",
      "/proc/timer_stats",
      "/proc/sched_debug",
      "/proc/scsi",
      "/sys/firmware",
      "/sys/devices/virtual/powercap"
    ],
    "ReadonlyPaths": [
      "/proc/bus",
      "/proc/fs",
      "/proc/irq",
      "/proc/sys",
      "/proc/sysrq-trigger"
    ]
  },
  "GraphDriver": {
    "Data": {
      "LowerDir": "/var/lib/docker/overlay2/80749fd6da97badee092e4be9eb30866bda47d6a5a5a13d956189758e30e51e1-init/diff:/var/lib/docker/overlay2/161cdc59f54708319cacf3aa112ddabe51fcf0b50f1d92b72f16a546b8afaf33/diff:/var/lib/docker/overlay2/1144b070160b974809d73662fea78ef5e924677bf550dedde81156ffd0eaf79c/diff:/var/lib/docker/overlay2/4107138fefaea9ba0aea85239c6900c3bb781b568ca2abb92925b130ac57fd10/diff:/var/lib/docker/overlay2/5aff5b7f5d1ee4d31ac8fec42793763b089e2a68f63dbd9b347522128c9bde7c/diff:/var/lib/docker/overlay2/13a9534601a222c4b6f826c43c8f9489d8e2a0b00c7d203b411c9dadfe1f263c/diff:/var/lib/docker/overlay2/6dd9c56bff29e74049f9d72565a9df582f0dfbf96aec4ea1da1e15b187c3fe2c/diff:/var/lib/docker/overlay2/3163c7c4fb63d810d819cb58dbd28cfca5ecd2c6ac7a09e5c08df03ba9cab97e/diff:/var/lib/docker/overlay2/35aad1222923b92b5c64de85d25912293d3565f3f242856fdd749b0a1b3e39b7/diff:/var/lib/docker/overlay2/ea1f116526201a0726c80c93c3dd283429251e973fe8992064aea72e32a4be74/diff:/var/lib/docker/overlay2/803ef911ed32357f9b67e1c52bf8efc898322a22fb972824d357e6971a5035e8/diff:/var/lib/docker/overlay2/42ef6c7466f3279520265d7d32170cb8445bb87dd5e53a192dc68ff0ca88a7c5/diff:/var/lib/docker/overlay2/9b13b6d9fbedd78af99fde58ed9cc289c240d2e1297cd7c9a586d4554866306e/diff:/var/lib/docker/overlay2/1645860b554c02c728717d906cce960092199f40636da76dd18833bedede5017/diff:/var/lib/docker/overlay2/f258a0d00805b77c2d11dc1e6a5ecefa26dc2824f24bdb85782a0c8f52fa0836/diff",
      "MergedDir": "/var/lib/docker/overlay2/80749fd6da97badee092e4be9eb30866bda47d6a5a5a13d956189758e30e51e1/merged",
      "UpperDir": "/var/lib/docker/overlay2/80749fd6da97badee092e4be9eb30866bda47d6a5a5a13d956189758e30e51e1/diff",
      "WorkDir": "/var/lib/docker/overlay2/80749fd6da97badee092e4be9eb30866bda47d6a5a5a13d956189758e30e51e1/work"
    },
    "Name": "overlay2"
  },
  "Mounts": [
    {
      "Type": "volume",
      "Name": "3f4b2b11cf5f1f97f05aa88cfa05d927b437831e9ca96cf1a2f28bd0677b083b",
      "Source": "/var/lib/docker/volumes/3f4b2b11cf5f1f97f05aa88cfa05d927b437831e9ca96cf1a2f28bd0677b083b/_data",
      "Destination": "/var/lib/postgresql/data",
      "Driver": "local",
      "Mode": "",
      "RW": true,
      "Propagation": ""
    }
  ],
  "Config": {
    "Hostname": "f904f1824139",
    "Domainname": "",
    "User": "",
    "AttachStdin": false,
    "AttachStdout": true,
    "AttachStderr": true,
    "ExposedPorts": {
      "5432/tcp": {}
    },
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "POSTGRES_INITDB_ARGS=--auth-host=scram-sha-256 --auth-local=scram-sha-256",
      "POSTGRES_USER=postgres",
      "POSTGRES_PASSWORD=u6SQrt!hSxB*0M4rcg2sRB",
      "POSTGRES_HOST_AUTH_METHOD=scram-sha-256",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/postgresql/17/bin",
      "GOSU_VERSION=1.17",
      "LANG=en_US.utf8",
      "PG_MAJOR=17",
      "PG_VERSION=17.0-1.pgdg120+1",
      "PGDATA=/var/lib/postgresql/data"
    ],
    "Cmd": [
      "postgres"
    ],
    "Image": "docker.io/library/postgres:17.0",
    "Volumes": {
      "/var/lib/postgresql/data": {}
    },
    "WorkingDir": "/",
    "Entrypoint": [
      "docker-entrypoint.sh"
    ],
    "OnBuild": null,
    "Labels": {
      "com.microsoft.developer.usvc-dev.build": "0.8.13",
      "com.microsoft.developer.usvc-dev.group-version": "usvc-dev.developer.microsoft.com/v1",
      "com.microsoft.developer.usvc-dev.lifecycle-key": "58286d90a49aee3d84664fd328e76dad604c62daecc37fd711fda4d9d501175b",
      "com.microsoft.developer.usvc-dev.name": "DefaultConnection-svfstknz",
      "com.microsoft.developer.usvc-dev.uid": "8c9532ee-887d-42fb-baa0-0319a337f976"
    },
    "StopSignal": "SIGINT"
  },
  "NetworkSettings": {
    "Bridge": "",
    "SandboxID": "aa57f7d6187858e5f62b7d642bf62c7b868a44ebde2c51abf31394a944e45da3",
    "SandboxKey": "/var/run/docker/netns/aa57f7d61878",
    "Ports": {
      "5432/tcp": [
        {
          "HostIp": "127.0.0.1",
          "HostPort": "32795"
        }
      ]
    },
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
      "default-aspire-network-sjeej4f2eg": {
        "IPAMConfig": {},
        "Links": null,
        "Aliases": [
          "DefaultConnection"
        ],
        "MacAddress": "02:42:ac:15:00:02",
        "NetworkID": "6b9ecfc2fae2f81e8bf2f1dd3814cf27253e0ee8e3bcfda7108c5f9a4c46a9d1",
        "EndpointID": "3333168b6401d02c106857602158b445d1e944a6b7e7781b7e46efbd4157f262",
        "Gateway": "172.21.0.1",
        "IPAddress": "172.21.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DriverOpts": {},
        "DNSNames": [
          "DefaultConnection-svfstknz",
          "DefaultConnection",
          "f904f1824139"
        ]
      }
    }
  }
}

That looks different to me than the following for the ports. Image

But it matches what VS containers says the ports for that image are: Image

Expected Behavior

That the dashboard starts up with healthy postgres and PG Admin.

Steps To Reproduce

var builder = DistributedApplication.CreateBuilder(args);
var postgres = builder.AddPostgres("DefaultConnection").WithPgAdmin();
<Project Sdk="Microsoft.NET.Sdk">
<Sdk Name="Aspire.AppHost.Sdk" Version="9.0.0" />
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <IsAspireHost>true</IsAspireHost>
    <UserSecretsId>86c4ff08-8351-4476-9dc2-fe1c048988e8</UserSecretsId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Aspire.Confluent.Kafka" Version="9.0.0" />
    <PackageReference Include="Aspire.Hosting.AppHost" Version="9.0.0" />
    <PackageReference Include="Aspire.Hosting.PostgreSQL" Version="9.0.0" />
    <PackageReference Include="Aspire.Hosting.Redis" Version="9.0.0" />
    <PackageReference Include="Aspire.Npgsql" Version="9.0.0" />
    <PackageReference Include="Aspire.Npgsql.EntityFrameworkCore.PostgreSQL" Version="9.0.0" />
    <PackageReference Include="Google.Protobuf" Version="3.28.3" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Abstractions" Version="9.0.0" />
    <PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="9.0.0" />
    <PackageReference Include="Npgsql" Version="9.0.1" />
    <PackageReference Include="Npgsql.NodaTime" Version="9.0.1" />
    <PackageReference Include="SixLabors.ImageSharp" Version="3.1.6" />
    <PackageReference Include="StackExchange.Redis" Version="2.8.16" />
    <PackageReference Include="System.Formats.Asn1" Version="9.0.0" />
    <PackageReference Include="Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore" Version="9.0.0" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Relational" Version="9.0.0" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="PostgresData\" />

  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\..\Infrastructure\Infrastructure.csproj" />
    <ProjectReference Include="..\..\Presentation\Presentation.csproj" IsAspireProjectResource="true" />
  </ItemGroup>

</Project>

Exceptions (if any)

No response

.NET Version info

.NET SDK: Version: 9.0.100 Commit: 59db016f11 Workload version: 9.0.100-manifests.c6f19616 MSBuild version: 17.12.7+5b8665660

Runtime Environment: OS Name: Windows OS Version: 10.0.22631 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\9.0.100\

.NET workloads installed: [aspire] Installation Source: VS 17.12.35514.174 Manifest Version: 8.2.2/8.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json Install Type: Msi

[maui-windows] Installation Source: VS 17.12.35514.174 Manifest Version: 9.0.0/9.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maui\9.0.0\WorkloadManifest.json Install Type: Msi

[maccatalyst] Installation Source: VS 17.12.35514.174 Manifest Version: 18.0.9617/9.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.maccatalyst\18.0.9617\WorkloadManifest.json Install Type: Msi

[ios] Installation Source: VS 17.12.35514.174 Manifest Version: 18.0.9617/9.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.ios\18.0.9617\WorkloadManifest.json Install Type: Msi

[android] Installation Source: VS 17.12.35514.174 Manifest Version: 35.0.7/9.0.100 Manifest Path: C:\Program Files\dotnet\sdk-manifests\9.0.100\microsoft.net.sdk.android\35.0.7\WorkloadManifest.json Install Type: Msi

Configured to use loose manifests when installing new manifests.

Host: Version: 9.0.0 Architecture: x64 Commit: 9d5a6a9aa4

.NET SDKs installed: 9.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 6.0.36 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 8.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 9.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

Learn more: https://aka.ms/dotnet/info

Download .NET: https://aka.ms/dotnet/download

Anything else?

No response

cisionmarkwalls commented 3 days ago

I also created a project from scratch. The only part I added was Pg and PgAdmin. Same unhealthy error out of the box. And note while running this that the redis cache is also marked as unhealthy.

Image

AspireApp1.zip

davidfowl commented 3 days ago

Does it stay unhealthy forever?

cisionmarkwalls commented 3 days ago

Yes, I left it running for about an hour. I'm wondering if it is Rancher Desktop, so I'm installing Podman and see if that makes a difference.

cisionmarkwalls commented 3 days ago

@davidfowl That is it. Rancher Desktop doesn't work with the healthchecks correctly. Podman does out of the box. Image

davidfowl commented 3 days ago

Ah this makes sense, we don’t support rancher desktop. I’ll add this to the list of rancher specific problems.

Looks like a networking issue with the health checks talking to the container for some reason.

afscrome commented 2 days ago

See https://github.com/dotnet/aspire/issues/6002#issuecomment-2414941401 - something between aspire 9.0 & rancher's networking does not play ball. You can work around this by setting .WithLifetime(ContainerLifetime.Persistent) on each and every container.