AKS no telemetry

[AnthonyDewhirst]

I have just plugged in the enricher into our microservices in AKS which have RBAC enabled.

We are using .net 6 and:

<PackageReference Include="Microsoft.ApplicationInsights.Kubernetes" Version="2.*" />
<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.21.0" />

We are using a cluster role and binding to enable this.

Here is a sample of the trace logging that we can see:

[Debug] [2022-12-12T08:52:37.2083354Z] Application Insights Kubernetes injected the service successfully.
[Debug] [2022-12-12T08:52:37.3053268Z] Kubernetes base address: https://10.0.0.1:443/
[Debug] [2022-12-12T08:52:37.3066981Z] Getting container id by environment variable ContainerId. Result: .
[Trace] [2022-12-12T08:52:37.3237449Z] [CGroupV1Matcher] No match for containerId. Input: 12:pids:/kubepods/besteffort/pod<guid>/<SomeId-1>, pattern: cpu.+/([^/]*)$
[Trace] [2022-12-12T08:52:37.3242237Z] [CGroupV1Matcher] No match for containerId. Input: 11:perf_event:/kubepods/besteffort/pod<guid>/<SomeId-1>, pattern: cpu.+/([^/]*)$
[Trace] [2022-12-12T08:52:37.3243998Z] [CGroupV1Matcher] Matched container id.
[Debug] [2022-12-12T08:52:37.3244854Z] [CGroupContainerIdProvider] Got container id by: 10:cpuset:/kubepods/besteffort/pod<guid>/<SomeId-1>
[Information] [2022-12-12T08:52:37.3245032Z] [CGroupContainerIdProvider] Got container id: <SomeId-1>
[Debug] [2022-12-12T08:52:37.3252482Z] Normalize container id: <SomeId-1>
[Trace] [2022-12-12T08:52:37.3253998Z] Container id normalized to: <SomeId-1>
[Debug] [2022-12-12T08:52:37.4426085Z] Initialize Application Insights for Kubernetes telemetry initializer with Options:
{"InitializationTimeout":"00:02:00","DisablePerformanceCounters":false}
[Trace] [2022-12-12T08:52:37.4543888Z] Creating K8sQueryClient
[Trace] [2022-12-12T08:52:37.4571018Z] Creating KubeHttpClient
[Debug] [2022-12-12T08:52:37.4615685Z] Access token is not null. Set default request header.
[Debug] [2022-12-12T08:52:37.4754718Z] Getting pod name by provider: UserSetPodNameProvider
[Debug] [2022-12-12T08:52:37.4756770Z] Try getting pod name by environment variable APPINSIGHTS_KUBERNETES_POD_NAME. Result: 
[Debug] [2022-12-12T08:52:37.4757038Z] Getting pod name by provider: HostNamePodNameProvider
[Debug] [2022-12-12T08:52:37.4757236Z] Try getting pod name by environment variable HOSTNAME. Result: <ms-name>-ef-migrations-8-drgwk
[Debug] [2022-12-12T08:52:37.4757530Z] Pod name by providers: <ms-name>-ef-migrations-8-drgwk
[Trace] [2022-12-12T08:52:37.4763497Z] Creating K8sQueryClient
[Trace] [2022-12-12T08:52:37.4763752Z] Creating KubeHttpClient
[Debug] [2022-12-12T08:52:37.4765352Z] Access token is not null. Set default request header.
[Trace] [2022-12-12T08:52:37.4791445Z] Default Header: Authorization: Bearer eyJhbGc...

[Trace] [2022-12-12T08:52:37.8236362Z] Not a authority signed certificate.
[Trace] [2022-12-12T08:52:37.8245960Z] Server Cert RAW: 
<cert-string>
[Trace] [2022-12-12T08:52:37.8250819Z] Building certificate chain.
[Trace] [2022-12-12T08:52:37.8251915Z] Client Cert RAW: 
<cert-string-2>
[Trace] [2022-12-12T08:52:37.8268339Z] Is Chain successfully built: True
[Trace] [2022-12-12T08:52:37.8526897Z] Query succeeded.
[Trace] [2022-12-12T08:52:37.8812972Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Information] [2022-12-12T08:52:37.9506300Z] Found pod by name providers: <ms-name>-ef-migrations-8-drgwk
[Debug] [2022-12-12T08:52:37.9514466Z] K8s pod info available in: 479 ms.
[Trace] [2022-12-12T08:52:37.9569193Z] Creating K8sQueryClient
[Trace] [2022-12-12T08:52:37.9569522Z] Creating KubeHttpClient
[Debug] [2022-12-12T08:52:37.9572289Z] Access token is not null. Set default request header.
[Trace] [2022-12-12T08:52:37.9573797Z] Default Header: Authorization: Bearer eyJhbGc...

[Trace] [2022-12-12T08:52:38.0034598Z] Not a authority signed certificate.
[Trace] [2022-12-12T08:52:38.0036076Z] Server Cert RAW: 
<cert-string-3>
[Trace] [2022-12-12T08:52:38.0036642Z] Building certificate chain.
[Trace] [2022-12-12T08:52:38.0036840Z] Client Cert RAW: 
<cert-string-4>
[Trace] [2022-12-12T08:52:38.0045691Z] Is Chain successfully built: True
[Trace] [2022-12-12T08:52:38.0255851Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2022-12-12T08:52:38.0511240Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2022-12-12T08:52:38.2197356Z] Query succeeded.
[Trace] [2022-12-12T08:52:38.2217136Z] Container status object: Microsoft.ApplicationInsights.Kubernetes.Entities.ContainerStatus, isReady: True
[Debug] [2022-12-12T08:52:38.2217742Z] K8s container info available in: 264 ms.
[Debug] [2022-12-12T08:52:38.2218586Z] Getting container status of container-id: <container-id>
[Trace] [2022-12-12T08:52:38.2365730Z] Default Header: Authorization: Bearer eyJhbGc...

[Trace] [2022-12-12T08:52:38.2692128Z] Not a authority signed certificate.
[Trace] [2022-12-12T08:52:38.2692839Z] Server Cert RAW: 
<cert-string-5>
[Trace] [2022-12-12T08:52:38.2693239Z] Building certificate chain.
[Trace] [2022-12-12T08:52:38.2693458Z] Client Cert RAW: 
<cert-string-6>
[Trace] [2022-12-12T08:52:38.2702566Z] Is Chain successfully built: True
[Trace] [2022-12-12T08:52:38.2816177Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2022-12-12T08:52:38.2853485Z] Application Insights for Kubernetes telemetry initializer is used but the content has not ready yet.
[Trace] [2022-12-12T08:52:38.2958260Z] Query succeeded.
[Trace] [2022-12-12T08:52:38.3099711Z] Default Header: Authorization: Bearer eyJhbGc...

and then many repeated rows of:

[Trace] [2022-12-12T08:52:44.1326589Z] Application Insights for Kubernetes telemetry initializer is invoked.
[Trace] [2022-12-12T08:52:44.1326997Z] Value is null or empty for key: Kubernetes.ReplicaSet.Name
[Trace] [2022-12-12T08:52:44.1327361Z] Value is null or empty for key: Kubernetes.Deployment.Name
[Trace] [2022-12-12T08:52:44.1327487Z] Finish telemetry initializer.

Looking in application inisghts for a trace record, I don't see any additional values:

I was under the impression that I would be able to see the values here, but I don't and the logging suggests that something isn't quite correct. All help would be appreciated. Thanks

[AnthonyDewhirst]

Actually I think that this was my bad. I had used the same name for all clusterRoleBindings and also misconfigured the logging within my c# app. I do have one thing though that has become apparent that might need putting in the documentation. When the c# app first starts and the Microsoft hosting logging kicks in. It appears that the k8s information hasn't been fetched yet and so isn't on the initial sets of telemetry. So what I was also seeing earlier when I first created this post is that the first sets of telemetry may indeed be missing the k8s information. Is this expected? I could see a scenario where someone new to this framework might apply it only having the hosting information for reference and then wonder why this telemetry hadn't been enriched.

[xiaomi7732]

Hi AnthonyDewhirst, thanks for the detailed info. Let me repeat what you said to make sure I understand the current situation correctly:

1. You are seeing K8s info now; but
2. There was missing info for the first few events/telemetry objects

For issue #2, it is by design. We have been trading that for reliable output.

That said, thank you for the reminder, I think it is worth documenting.

[xiaomi7732]

I have updated the FAQ. I'll close this issue for now. Feel free to reopen or file new ones if there's any follow-up needed.