Closed TTmaister closed 1 year ago
Hi @TTmaister, this is a known issue because of the implementations of docker engines. There has been a related issue #290.
Please check out the design doc here for the container id providers: https://github.com/microsoft/ApplicationInsights-Kubernetes/wiki/Design-for-ContainerIdProviders
That said, 2 questions for you:
Is the container id anywhere in the mountinfo
you shared? If it is, we might be able to append another container id providers for your case.
Is it possible for you to set up the environment variable for container id as mentioned in the design doc to workaround the issue? For example: ContainerId=my-container-id
.
Any proposal for how to make this more adaptive is super-welcome!
BTW, just FYI, here's a lengthy stackoverflow discussion on the topic: https://stackoverflow.com/questions/20995351/how-can-i-get-docker-linux-container-information-from-within-the-container-itsel/72565733#72565733
And an open standard issue:
Container ID was 34bc656c1739fe713a867e18cc307f96e6fcba3cf89755db31b3c837f9d137f5. The answer is that there is no container id visible in mountinfo.
Possibly I can set the container id in the environment variables, but I don't know how to do it in the Kubernetes Deployment yaml so that each pod has its own ID. Does it have to be the same as what the Kubernetes Api provides.
Hey @TTmaister, thanks for providing the information. Unfortunately, without container id anywhere in the container, we won't have the magic to somehow make it available.
Manually set the environment would probably be the only way that will provide full enhancement. I don't know what will happen if you set container id to a mismatched value. That is an interesting idea. I assume you could set the environment variable to a random value for identification of the instances, but will it provide too much value? (pods comes and goes, and if the container id is random, why bother?)
That said, could you please help me understand what the telemetry looks like on your end? I assume container id missing but everything else, like podName
, node
info should be there on the events, does that align with what you see?
Application Insight is missing all the data brought by the plugin. I can see in Application Insight that the POD requests information about the POD from the Kubernetes API and gets the response http200.
Hey @xiaomi7732
Does it make any sense to change the ContainerIdHolder class to be able to parse the ContainerName variable from the environment variables.
And then filtter ContainerStatus by ContainerName
if (containerStatuses is not null && containerStatuses.Count > 1)
{
string? containerName = Environment.GetEnvironmentVariable("ContainerName");
containerStatus = containerStatuses.FirstOrDefault(c => c.Name == containerName);
_logger.LogInformation(FormattableString.Invariant($"Use the only container inside the pod for container id: {containerStatus.ContainerID}"));
using (IServiceScope scope = _serviceScopeFactory.CreateScope())
{
IContainerIdNormalizer normalizer = scope.ServiceProvider.GetRequiredService<IContainerIdNormalizer>();
if (normalizer.TryNormalize(containerStatus.ContainerID, out string? normalizedContainerId))
{
_containerId = normalizedContainerId;
return true;
}
}
_logger.LogError(FormattableString.Invariant($"Normalization failed for container id: {containerStatus.ContainerID}"));
}
ContinerName can be defined as environment variable Kubernetes Deployment yaml.
env:
- name: ContainerName
valueFrom:
fieldRef:
fieldPath: metadata.labels['app']
Or some other label. We have set "app" label for all deployments.
Hi @TTmaister, using ContainerName
for matching seem like an approach. I think we should peruse it when time permits. Do you know is pod name always unique?
With regarding what is going on, I think I see it. Container id was supposed to be optional, but it is required - in between several iterations to support .NET 6 and the latest K8s SDK. See below for the details:
It was like this, there had been a note to allow empty container id.
// Notes: It is still possible for the optional container id to be empty at this point, the following method needs to handle the case.
if (!await SpinWaitContainerReadyAsync(timeoutAt, queryClient, myPod, containerId, cancellationToken).ConfigureAwait(false))
{
_logger.LogError(Invariant($"Kubernetes info is not available before the timeout at {timeoutAt}."));
return null;
}
And in the implementation, this is the logic to allow empty container id:
if (!string.IsNullOrEmpty(myContainerId))
{
// Check targeted container status
readyToGo = IsContainerReady(podInfo.GetContainerStatus(myContainerId));
}
else
{
_logger.LogWarning("No container id available. Fallback to use the any container for status checking.");
readyToGo = podInfo.GetAllContainerStatus().Any(s => IsContainerReady(s));
}
And it is like this now, meaning optional container id is not allowed:
public async Task<bool> IsContainerReadyAsync(CancellationToken cancellationToken)
{
V1ContainerStatus? myContainerStatus = await GetMyContainerStatusAsync(cancellationToken).ConfigureAwait(false);
if (myContainerStatus is not null)
{
return IsContainerStatusReady(myContainerStatus);
}
return false;
}
That doesn't agree with the design that I shared earlier and shall be treated as a bug. Will it unblock you if I make it optional again?
Pod can be recreate with the same name, but there can only be one with the same name at a time. Object Names and IDs
If I tested it right... if Container Id missing then Application Insight not showing "Cloud Role Name". But is show and populate this custom properties
Kubernetes.Node.Name Kubernetes.Pod.Name Kubernetes.Pod.Namespace Kubernetes.Node.ID Kubernetes.Deployment.Name Kubernetes.Pod.ID Kubernetes.ReplicaSet.Name Kubernetes.Pod.Labels
I think there could be 2 fixes:
And I inspected various options to back fill container id, and I think using container name is a very good idea. It might worth a bit documentation but could be useful as an alternative to container ids.
@TTmaister are you interested to submit a PR for it? Or I can prepare a PR if you don't have the time.
BTW, if you are interested, please fork this repository, and branching off your own fork since this repository has been locked down and nobody could PR into it directly. :-)
@TTmaister your change has been released:
https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1-beta2
Let me know if that unblocks you. Thank you for your contribution!
@TTmaister your change has been released:
https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1-beta2
Let me know if that unblocks you. Thank you for your contribution!
Works as expected.
@xiaomi7732 Maybe this should be documented to project readme.
@TTmaister, thanks for the verification! I will update the wiki. I'll keep this issue open until the stable version of 6.1.1 got released.
A FAQ is added: https://github.com/microsoft/ApplicationInsights-Kubernetes/wiki/FAQ, pointing to the wiki: How to set container name manually.
Is there an Eta on when the version 6.1.1 will be released?, Thx!
Hey @Ismael-Pep, thanks for the inquiry. Here's the new package is released today: https://www.nuget.org/packages/Microsoft.ApplicationInsights.Kubernetes/6.1.1
After upgrade AKS 1.23.x to 1.25.x Application Insight Kubernetes addon can't get container Id if multiple container are used.
RBAC roles and binding has been same before upgrade.
ContainerIdProviders not