search

microsoft / azure-container-apps

Roadmap and issues for Azure Container Apps
MIT License
361 stars 29 forks source link

DAPR endpoint for checking outbound health of sidecar returning NOTFOUND #460

Open lurock opened 1 year ago

lurock commented 1 year ago

Please provide us with the following information:

This issue is a: (mark with an x)

Issue description

In ASP.NET Core there is an extension method provided by the DAPR SDK called "AddDaprSecretStore" that waits for the side car to be ready by checking the health using "http://localhost:3500/v1.0/healthz/outbound" and when that returns true. It then does a bulk load of your secrets from your secret store DAPR component into your ASP.NET configuration. This is not working because the endpoint "http://localhost:3500/v1.0/healthz/outbound" is returning a NotFound error. I have added a link to the DAPR SDK source code where it calls that endpoint: https://github.com/dapr/dotnet-sdk/blob/3224579c3e136020bbf713528965854ebbd4eefe/src/Dapr.Client/DaprClientGrpc.cs#L1339

Steps to reproduce

  1. Bring up and app and call into the side car using http://localhost:3500/v1.0/healthz/outbound
  2. Returns a NotFound error.

Expected behavior

I would like to be able to use the DotNet DAPR SDK to add my secrets to the ASP.Net core configuration using the extension method provided "AddDaprSecretStore".

Actual behavior

The extension method times out and secrets are not added to the configuration.

kendallroden commented 1 year ago

As mentioned via email, checking on this and will report back

ItalyPaleAle commented 1 year ago

Hi @lurock thanks for reporting this issue! We've identified the root cause as a bug in Dapr with the way it's used in Azure Container Apps. We are working on a mitigation and I'll be able to share with you a timeline shortly.

ItalyPaleAle commented 1 year ago

Hi @lurock, rollout of the fix should begin in the next days and, assuming everything goes well (as it should! :) ), should be available in all regions within a couple of weeks.

If you can share what Azure region you're using for your Container Apps, I can make a note to let you know when it's upgraded

lurock commented 1 year ago

@ItalyPaleAle thank you. We are west us.

coreenoliva commented 1 year ago

Hi @ItalyPaleAle & @kendallroden, I'm seeing a similar issue when trying to use AddDaprSecretStore and WaitForSidecarAsync from the Dapr SDK as well when deployed to Container Apps where the call eventually times out.

Turning on debug logs for Dapr gives me this -

image

Was wondering if this is also a symptom of the issue that was fixed?

ItalyPaleAle commented 1 year ago

@coreenoliva Yes that look like the same issue.

The rollout will begin tomorrow (Friday) and we expect all Azure regions to have the fix complete in about 2 weeks.

digitalkingsley commented 1 year ago

@coreenoliva Yes that look like the same issue.

The rollout will begin tomorrow (Friday) and we expect all Azure regions to have the fix complete in about 2 weeks.

This fix would be greatly appreciated, as I've been stuck for days, because of it, trying to read specific secrets (with DaprSecretDescriptor) from Azure Key Vault during startup, by calling AddDaprSecretStore.

In the interim, I'm considering reading those secrets from environment-variables (less secure) during startup, as opposed to directly from the vault, pending when this issue is fixed.

For the record, my region is East US. I'd also appreciate being alerted to the update, @ItalyPaleAle.