Watch task over kubernetes Endpoint resources terminated unexpectedly

[ys0d]

Running a starter aspire solution I get the following error.

crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
      Watch task over kubernetes Container resources terminated unexpectedly. Check to ensure dcpd process is running.
      System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://webproxy.xxxxxxxx.net:8080/' failed with status code '403'."
         at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
         at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
         at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
         at k8s.WatcherExt.<>c__DisplayClass1_0`2.<<MakeStreamReaderCreator>b__0>d.MoveNext()
      --- End of stack trace from previous location ---
         at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
         at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
      --- End of stack trace from previous location ---
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
      --- End of stack trace from previous location ---
         at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
         at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
         at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
         at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
      --- End of stack trace from previous location ---
         at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
      --- End of stack trace from previous location ---
         at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
      --- End of stack trace from previous location ---
         at Polly.Outcome`1.GetResultOrRethrow()
         at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
         at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_0.<<WatchResourceChanges>g__WatchKubernetesResource|3>d`1.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 281

I have created the solution from the visual studio aspire template and I haven't made any changes.
What could be done to fix it?

[davidfowl]

cc @karolz-ms @danegsta

[danegsta]

@ys0d that error seems to indicate that you have a proxy configured that’s returning errors for Aspire traffic. That 403 error code indicates a forbidden request (usually due to missing order incorrect credentials).

[ys0d]

@danegsta I have checked proxy configuration and it looks I don't have one.

PS C:\Users\YS0D> netsh winhttp show proxy

Current WinHTTP proxy settings:

    Direct access (no proxy server).

But i work under a VPN connection, which maybe causes that problem. The question is how to fix it? Which settings have to be adjusted and permission granted?

[danegsta]

It's entirely possible the VPN/proxy (the connection is definitely going through a proxy server as the error message you're getting is a low level .NET proxy connection error, not something specific to Aspire) doesn't support self-signed certificates. By default, Aspire uses https traffic on localhost using the .NET dev cert, which is a per-machine self-signed certificate restricted to localhost.

You may want to try running the http launch profile (instead of the default https profile). For Visual Studio, you can simply change the target launch profile dropdown. From the command line, you'll need to add --launch-profile http to your run command arguments.

[ys0d]

Thanks for explanations. After an investigation I can agree it goes through a proxy, probably because of ipv6 is being used. Looking at PAC file I see that 127.* addresses are configured to go directly but it is not the case for ::1 addresses. Is it possible to configure Aspire services to use ipv4?

using an http profile produces the same result:

info: Aspire.Hosting.DistributedApplication[0]
      Login to the dashboard at http://localhost:15261/login?t=407234c531f4a953cceefb6e2f749509
crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
      Watch task over kubernetes Service resources terminated unexpectedly. Check to ensure dcpd process is running.
      System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://webproxy.xxxxxx.net:8080/' failed with status code '403'."

[danegsta]

Not in the current 8.0 GA release, but in main branch support was added for an environment variable to control IP binding preferences. Setting DCP_IP_VERSION_PREFERENCE=ipv4 will cause ipv4 to be preferred for any service bindings. This will ship with the upcoming 8.1 Aspire release.

[dbreshears]

This appears to be done in 8.1, going to close, re-open if the issue continues after acquiring 8.1