search

microsoft / ApplicationInsights-Kubernetes

Enrich the telemetry data for .NET applications running inside containers that are managed by Kubernetes.
Other
135 stars 57 forks source link

Cannot get pod info because of proxy configuration #351

Closed henols02-sr closed 1 year ago

henols02-sr commented 1 year ago

Describe the bug My ASP.NET Core 7 application logs the following warnings:

[Warning] [2023-05-08T12:59:05.7419250Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T12:59:36.9033456Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T13:00:07.9161911Z] Query exception while trying to get pod info: The proxy tunnel request to proxy 'http://proxy:8080/' failed with status code '502'."
[Warning] [2023-05-08T13:00:34.4382518Z] Query exception while trying to get pod info: A task was canceled.
[Critical] [2023-05-08T13:00:34.4409067Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)

The reason for this is that I have set the HTTP_PROXY and HTTPS_PROXY environment variables. What URL (or rather host name) is this library using for getting pod info? I want to set the NO_PROXY environment variable because the proxy is only for connections to the Internet, and we have an on-premise installation of Kubernetes. Currently NO_PROXY is set to localhost,127.0.0.1,.local and a couple of more.

Package Versions Application Insights Kubernetes Version: 6.1.0 Application Insights SDK Version: 2.21.0

xiaomi7732 commented 1 year ago

Hey @henols02-sr, thank you for the reporting. Let me check the code and get back to you.

xiaomi7732 commented 1 year ago

Here's the environment variable for the host name, could you please look it up in your environment?

KUBERNETES_SERVICE_HOST
KUBERNETES_SERVICE_PORT

And the default values are: Default host: kubernetes.default.svc Default port: 443

Here's the related source code:

https://github.com/kubernetes-client/csharp/blob/1342531db153b48d315921bdb1d1821174d36ebb/src/KubernetesClient/KubernetesClientConfiguration.InCluster.cs#LL51C1-L52C1

henols02-sr commented 1 year ago 
/app # env | grep KUBERNETES
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_SERVICE_HOST=10.43.0.1

So the address I'm looking for seems to be 10.43.0.1. I'll add that to NO_PROXY and see how it goes.

xiaomi7732 commented 1 year ago

@henols02-sr, please try it and let us know.

henols02-sr commented 1 year ago

Now I don't get the bit about proxy but it still doesn't work:

[Warning] [2023-05-11T12:05:49.4480166Z] Query exception while trying to get pod info: A task was canceled.
[Warning] [2023-05-11T12:06:09.0392950Z] Query exception while trying to get pod info: A task was canceled.
[Critical] [2023-05-11T12:06:09.0419462Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)

I will talk to our Kubernetes team.

xiaomi7732 commented 1 year ago

@henols02-sr, if you turn on trace logging by following this wiki, you shall be able to see the issue: https://github.com/microsoft/ApplicationInsights-Kubernetes/blob/develop/docs/SelfDiagnostics.MD#with-204-beta1-or-above

Please give it a shot and let me know if there's anything unexpected.

henols02-sr commented 1 year ago

I turned on tracing and here is the result:

[Trace] [2023-05-15T14:13:18.4686126Z] Registering bootstrap and hosted service.
[Information] [2023-05-15T14:13:18.5642140Z] Skip registering K8sInfoBackgroundService by user configuration.
[Trace] [2023-05-15T14:13:18.5647911Z] Registered bootstrap and hosted service.
[Debug] [2023-05-15T14:13:18.5649481Z] Application Insights Kubernetes injected the service successfully.
[Trace] [2023-05-15T14:13:20.5597187Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:37.5685781Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.6616908Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.6646638Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:39.9642832Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0642293Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0664903Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.0677772Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.1670951Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3598338Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3776381Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.3777975Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:40.4712580Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.2625747Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.5603021Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.6706579Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:41.6716769Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:42.3633168Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:42.5609480Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Debug] [2023-05-15T14:13:43.6646719Z] Starting update K8sEnvironment
[Debug] [2023-05-15T14:13:44.8601395Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:13:44.8604740Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:13:44.8605481Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:13:44.9629228Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:13:44.9629830Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:13:45.7629870Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.0608595Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1595579Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1602106Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:46.1604096Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:47.8641744Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:47.9634433Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:49.8661289Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:50.0629243Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3613144Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3624510Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.3632592Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.8597378Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:56.8603084Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.0611344Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.0622455Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.3604841Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:13:57.4607701Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0662887Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0667259Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0668279Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0668768Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669170Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669484Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669733Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0669964Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0670244Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:01.0670494Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:03.8100580Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:03.8100577Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:13.8081906Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:13.8081906Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3627769Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3662496Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:21.3682503Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:23.8080337Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:23.8081450Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:33.8077301Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:33.8078547Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:40.8655909Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:43.8075426Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:43.8076482Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:53.8073731Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:14:53.8073535Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:00.9995484Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:00.9996393Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:03.8076598Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:03.8076672Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:13.8082821Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:13.8082871Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3587711Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3591438Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:21.3593114Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:23.8071272Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:23.8071744Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:25.0646879Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:25.0658901Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Warning] [2023-05-15T14:15:25.0686069Z] Query exception while trying to get pod info: A task was canceled.
[Trace] [2023-05-15T14:15:25.0759046Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedPodWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CoreV1OperationsExtensions.ReadNamespacedPodAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.GetMyPodAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
[Debug] [2023-05-15T14:15:25.5754421Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:15:25.5755023Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:15:25.5755072Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:15:25.5755110Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:15:25.5755131Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:15:33.8073760Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:33.8073946Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:40.8658556Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:43.8073109Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:43.8073109Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:44.6588611Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:44.6589131Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Warning] [2023-05-15T14:15:44.6596068Z] Query exception while trying to get pod info: A task was canceled.
[Trace] [2023-05-15T14:15:44.6605610Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
   at k8s.AbstractKubernetes.k8s.ICoreV1Operations.ReadNamespacedPodWithHttpMessagesAsync(String name, String namespaceParameter, Nullable`1 pretty, IReadOnlyDictionary`2 customHeaders, CancellationToken cancellationToken)
   at k8s.CoreV1OperationsExtensions.ReadNamespacedPodAsync(ICoreV1Operations operations, String name, String namespaceParameter, Nullable`1 pretty, CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.GetMyPodAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
[Critical] [2023-05-15T14:15:44.6618284Z] System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.ApplicationInsights.Kubernetes.Pods.PodInfoManager.WaitUntilMyPodReadyAsync(CancellationToken cancellationToken)
   at Microsoft.ApplicationInsights.Kubernetes.K8sEnvironmentFactory.CreateAsync(CancellationToken cancellationToken)
[Debug] [2023-05-15T14:15:44.6640928Z] Finished update K8sEnvironment, next iteration will happen at 05/15/2023 14:15:46 (UTC) by interval settings of 00:00:02
[Debug] [2023-05-15T14:15:46.6628421Z] Starting update K8sEnvironment
[Debug] [2023-05-15T14:15:46.6643397Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T14:15:46.6643686Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2023-05-15T14:15:46.6643716Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2023-05-15T14:15:46.6643754Z] Try getting pod name by environment variable HOSTNAME. Result: abc-backend-74d4fcc97c-7km4w
[Debug] [2023-05-15T14:15:46.6643778Z] Pod name by providers: abc-backend-74d4fcc97c-7km4w
[Trace] [2023-05-15T14:15:53.8076062Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:15:53.8076068Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9996989Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997396Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997495Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:00.9997613Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:03.8078271Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:03.8078272Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:13.8074145Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:13.8077361Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3587918Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3592904Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:21.3596254Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:23.8075679Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:23.8080065Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:33.8079620Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:33.8079621Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:40.8657275Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:43.8077016Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:43.8077015Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:53.8076268Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:16:53.8076265Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:01.0000398Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:01.0000963Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:03.8067795Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:03.8067793Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:13.8070421Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2023-05-15T14:17:13.8070420Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
xiaomi7732 commented 1 year ago

Hi @henols02-sr, the logs showed that the requests timed out when querying the cluster. There are no details revealing why. Can you search your application insights logs for exceptions and see if you could correlate it to requests and find out why it timeout? Maybe, do you have a firewall to prevent requests into the cluster (from within the container)?

On my side, I did a simple experiment and I think the configuration should work. That is why I think there might be another security mechanism like a firewall or so.

Here are the details:

  1. Setup proxy by:
ENV HTTP_PROXY=http://proxy:8080/
ENV HTTPS_PROXY=http://proxy:8080/

And I got the following expected error since I don't have a real proxy server

[Warning] [2023-05-15T18:35:15.6404868Z] Query exception while trying to get pod info: Resource temporarily unavailable (proxy:8080)  # I don't have a real proxy
[Debug] [2023-05-15T18:35:16.1785030Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2023-05-15T18:35:16.1786398Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result:
...```

And then, I exclude my cluster:

```shell
ENV NO_PROXY=10.96.0.1

And after that, it worked again for me.

Here's my cluster environment:

PS D:\Repos> kubectl exec -it $pod -- /bin/bash
root@ai-k8s-f5webapi-deployment-77dc74c9dc-j4qqf:/App# printenv
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
HOSTNAME=ai-k8s-f5webapi-deployment-77dc74c9dc-j4qqf
DOTNET_VERSION=6.0.16
ASPNETCORE_URLS=http://+:80
PWD=/App
HOME=/root
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
APPLICATIONINSIGHTS__CONNECTIONSTRING=InstrumentationKey=5d8258e7-abb2-4066-89a5-8c73071b74ff;IngestionEndpoint=https://westus2-0.in.applicationinsights.azure.com/;LiveEndpoint=https://westus2.livediagnostics.monitor.azure.com/
TERM=xterm
SHLVL=1
ASPNET_VERSION=6.0.16
HTTPS_PROXY=http://proxy:8080/
HTTP_PROXY=http://proxy:8080/
KUBERNETES_PORT_443_TCP_PROTO=tcp
DOTNET_RUNNING_IN_CONTAINER=true
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1 # I added this to the no_proxy list
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/printenv
henols02-sr commented 1 year ago

Yes, there is probably some network policy blocking the request. I will work the Kubernetes team to get to bottom if it.