davidfowl
commented
6 months ago Looks like your VPN blocks ipv6 localhost?
Closed afscrome closed 2 months ago
davidfowl
commented
6 months ago Looks like your VPN blocks ipv6 localhost?
afscrome
commented
6 months ago Yup - that seems to be the issue - ping localhost -6 fails when not connected to the VPN.
Unfortunately after speaking to my IT team, they have no plans to make any changes to fix this - is it possible to have aspire connect / fall back to ipv4 localhost instead?
davidfowl
commented
6 months ago @karolz-ms can you detect this scenario?
karolz-ms
commented
6 months ago What we can do is to have a policy preference for IPv4 or IPv6. Probably default to IPv4. I am going to open an issue about that in DCP repo.
ElanHasson
commented
4 months ago I have hit similar to this but no VPN.
davidfowl
commented
4 months ago What issue exactly?
redstrike
commented
4 months ago Aspire starter templates work fine on my home PC. However, when I try to follow the quickstart guide at my company workstation, it seems to be blocked by the proxy. At first, I thought it was related to K8s starting issues with Rancher Desktop. Now, I notice the errors occurred even with a barebone AppHost (dotnet new aspire -o AspireHello), which doesn't create any OCI containers.
PS C:\Users\tnguyenthanh\AspireHello> dotnet run --project AspireHello.AppHost
Building...
info: Aspire.Hosting.DistributedApplication[0]
Aspire version: 8.0.1+a6e341ebbf956bbcec0dda304109815fcbae70c9
info: Aspire.Hosting.DistributedApplication[0]
Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
Application host directory is: C:\Users\tnguyenthanh\AspireHello\AspireHello.AppHost
info: Aspire.Hosting.DistributedApplication[0]
Now listening on: https://localhost:17218
info: Aspire.Hosting.DistributedApplication[0]
Login to the dashboard at https://localhost:17218/login?t=b44ba04fddf89593dbacc5440258be31
crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
Watch task over kubernetes Container resources terminated unexpectedly. Check to ensure dcpd process is running.
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'BadGateway', response body
... (hidden due to too long)
<!--Info-->
<table class="infoTable">
<tr>
<td class="infoData">
<b>URL: </b><script type="text/javascript">break_line("https://[::1]:19885/apis/usvc-dev.developer.microsoft.com/v1/containers?watch=true");</script><br />
</td>
</tr>
</table>
<!--/Info-->
... (too long, please check the attachment)
at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.WatcherExt.<>c__DisplayClass1_0`2.<<MakeStreamReaderCreator>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
--- End of stack trace from previous location ---
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
--- End of stack trace from previous location ---
at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_0.<<WatchResourceChanges>g__WatchKubernetesResource|3>d`1.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 281
ys0d
commented
3 months ago I am experiencing the same issue running any Aspire starter project in my company's VPN environment...
info: Aspire.Hosting.Dcp.dcpctrl[0]
starting controller manager
crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
Watch task over kubernetes Executable resources terminated unexpectedly. Check to ensure dcpd process is running.
System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://webproxy.xxxxxxx.net:8080/' failed with status code '403'."
at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync```
why is that happenning? Where does app host try to reach through a tunnel?Tried to mitigate this by setting DOTNET_SYSTEM_NET_DISABLEIPV6 environment to true, but that then fails with another error is it seems to be trying to force the use of IPv6 somewhere along the line.
crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
Watch task over kubernetes Executable resources terminated unexpectedly. Check to ensure dcpd process is running.
System.Net.Http.HttpRequestException: The requested name is valid, but no data of the requested type was found. ([::1]:57540)
---> System.Net.Sockets.SocketException (11004): The requested name is valid, but no data of the requested type was found.
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecutionContextCallback(Object s)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.ExecuteFromThreadPool(Thread threadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.ICustomObjectsOperations_ListClusterCustomObjectWithHttpMessagesAsync[T](String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.k8s.ICustomObjectsOperations.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
at k8s.WatcherExt.<>c__DisplayClass1_0`2.<<MakeStreamReaderCreator>b__0>d.MoveNext()
--- End of stack trace from previous location ---
at k8s.Watcher`1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task`1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+MoveNext()
at k8s.Watcher`1.CreateWatchEventEnumerator(Func`1 streamReaderCreator, Action`1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+MoveNext() in /_/src/Aspire.Hosting/Dcp/KubernetesService.cs:line 209
at Aspire.Hosting.Dcp.KubernetesService.WatchAsync[T](String namespaceParameter, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult()
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
--- End of stack trace from previous location ---
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_1`1.<<WatchResourceChanges>b__10>d.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 285
--- End of stack trace from previous location ---
at Polly.ResiliencePipeline.<>c.<<ExecuteAsync>b__3_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync(Func`2 callback, CancellationToken cancellationToken)
at Aspire.Hosting.Dcp.ApplicationExecutor.<>c__DisplayClass32_0.<<WatchResourceChanges>g__WatchKubernetesResource|3>d`1.MoveNext() in /_/src/Aspire.Hosting/Dcp/ApplicationExecutor.cs:line 281@afscrome it is not so much "forcing IPv6", but rather a preference for IPv6 if both IPv6 and IPv4 are available. I am thinking we need a way to turn that preference in the opposite direction.
karolz-ms
commented
2 months ago This is going to be fixed in 8.1 release. We are introducing a new environment variable, DCP_IP_VERSION_PREFERENCE, that determines which version of the IP protocol will be preferred for communicating with DCP and for allocating service ports. The value can be set to IPv4, v4 or 4 (to prefer IP protocol version 4) or IPv6, v6 or 6 (to prefer IP protocol version 6).
The preference determines which type of available IP addresses will be preferred when resolving localhost host name. localhost is the default pseudo-address that programs, containers and DCP itself bind to. If a program or container is using some different host name or address, the DCP_IP_VERSION_PREFERENCE environment variable has no effect--the name will be resolved using default DNS resolver mechanism and the resulting address(es) will be tried.
ys0d
commented
2 months ago Tried this new option DCP_IP_VERSION_PREFERENCE with IPv4 value. Unfortunately it did solve the issue. Still, for unknown reason, host tries to go somewhere through a proxy... The following log is from running an empty Aspire app created from a VS template.
info: Aspire.Hosting.DistributedApplication[0]
Aspire version: 8.0.2+3d0cf817f24298ba37bca94f15bd2f7061856e77
info: Aspire.Hosting.DistributedApplication[0]
Distributed application starting.
info: Aspire.Hosting.DistributedApplication[0]
Application host directory is: C:\Temp\aspire\aspire.AppHost
info: Aspire.Hosting.DistributedApplication[0]
Now listening on: https://localhost:17277
info: Aspire.Hosting.DistributedApplication[0]
Login to the dashboard at https://localhost:17277/login?t=9639615b7c2b1fac7b588e4b46028ab0
crit: Aspire.Hosting.Dcp.ApplicationExecutor[0]
Watch task over kubernetes Container resources terminated unexpectedly. Check to ensure dcpd process is running.
System.Net.Http.HttpRequestException: The proxy tunnel request to proxy 'http://webproxy.***.net:8080/' failed with status code '403'."
at System.Net.Http.HttpConnectionPool.EstablishProxyTunnelAsync(Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)Or I didn't get it right and this setting is not respected yet in the 8.0.2/8.0.100 version?
karolz-ms
commented
2 months ago @ys0d you can verify whether you have the correct DCP version by running dcp version from the command line (you might need to find where that exe lives in the dotnet packs folder). The version that supports DCP_IP_VERSION_PREFERENCE environment variable is 0.5.6 or newer.
That said, I assume you have a system-wide proxy configured for HTTP connections. That is independent from the decision whether "localhost" will translate into IPv4 or IPv6 address. If you do not wish to use the proxy at all, you will have to disable it, at least for your Aspire application run.
Hope this helps!
ys0d
commented
2 months ago @karolz-ms you were right... I wasn't very transparent because I could reach localhost resources via browser, but in some cases failed using other tools... What I found is that we have http(s)_proxy environment variables set and they are respected by some tools. Crearing them completely solves that issue, but we need them to be set. I had tried to put exceptions for localhost addresses using NO_PROXY env var, but it didn't help. Looks like aspire tools respect HTTP_PROXY but ignore NO_PROXY. Would be more useful to be able to use both (like curl, for example, do) or none...
karolz-ms
commented
2 months ago @ys0d NO_PROXY should work, at least according to https://pkg.go.dev/golang.org/x/net/http/httpproxy, which is what DCP is using under the covers... 🤔
Maybe you can play with different values, specific IP addresses etc. and see if some combination works?
ys0d
commented
2 months ago @karolz-ms
I tried this combination and it didn't work.
NickHodgeBB
commented
1 month ago I found this helped me find what ports were reserved ... moved the launchSettings.json for http and https free ports and my ASPNET Core app within Aspire launched successfully.
netsh interface ipv4 show excludedportrange protocol=tcp
If I try to start an aspire app when I have no network connection, it fails to start with the below error. The same error happens both when starting through Visual Studio, and
dotnet runAdditionally, when starting in Visual Studio (Version 17.10.0 Preview 2.0), it pops up the following dialog box which before the Aspire app even starts
My company does have an always on VPN installed, which blocks internet connectivity when not connected to the VPN, so it's possible that is interfering here. However even with that, I can still normally open and connect to ports running on localhost.
VS Info
dotnet workloads
Aspire version in AppHost csproj