Windows Containers Observability

[riyapatel-ms]

Discussed in https://github.com/microsoft/Windows-Containers/discussions/310

^{Originally posted by **riyapatel-ms** January 31, 2023} Hey all, The Windows Containers team has taken a focus on its investments in observability offerings and practices. As such, we'd like to hear more about the community's use-cases and current observability set-ups, whether it may be through Container Insights or third-party solutions such as Prometheus, Grafana, etc. We'd also like to hear about any concerns or features you'd like to see. Is there clear functionality missing in your observability story? Do you have any concerns? Is there anything that performs well that you'd like to see grow? Any input would be valuable in our ongoing discussions.

[mloskot]

Yes! Please, help out about logs collection, make it possible and easily as it is for Linux nodes/pods.

For example, apparently, the Filebeat does not collect logs from Windows pods

From https://github.com/elastic/beats/issues/16814

Running Beats as proposed now in Kubernetes guides is not possible there because this requires to run Beats as containers, but so far Beats containers are only created for Linux.

From https://discuss.elastic.co/t/windows-containers-logging/221997

Filebeat is also supported in Windows but I guess that the main difference would be that Beats are not distributed as Windows containers yet.

For Windows containerised applications, logs collection requires some non-trivial gymnastics, some invaluable notes here https://gist.github.com/jsturtevant/73b0bfe301a6abecd951b6f98bddffd4, so it would be very helpful if Microsoft offered best practices and solutions for most common situations.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[mloskot]

Please, don't close. This issue is very important to get addressed.BTW, @riyapatel-ms, would there be any updates?

[riyapatel-ms]

@mloskot I don't plan on closing this issue anytime soon, so feel free to use this as a dump as you run into more limitations/suggestions. In terms of updates, we plan to update our documentation and release a "best practices/recommendations" soon.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[mloskot]

no close

[adeturner]

The lack of a cadvisor equivalent for windows is limiting.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[mloskot]

do not close, unless anyone from MSFT offer conclusive comments

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @riyapatel-ms, please provide an update or close this issue.

[adeturner]

keep alive, pending MS update

[riyapatel-ms]

Pinging to keep issue open, this workstream has been transferred to another, will update once I have her GitHub

[connexallcloud]

Please don't close this thread, policy bot!

I'm also interested to see what can be improved with Windows container observability. We are currently using Datadog and Kubecost inside an EKS Kubernetes cluster, and have found issues with the level of detail that either can gather regarding Windows CPU and memory utilization versus what they can achieve with Linux workloads. Apparently cadvisor can't come up with the goods. Indeed, for many of these cloud services, it seems that Windows containers are almost a second class citizen compared to Linux support. Therefore any advancements that can be made as suggested by @adeturner and others to bring some level of obervability parity would be fantastic.

[fady-azmy-msft]

What sort of best practices are you looking for? And in what situations do you see there's missing documentation? Any insight on this would help us prioritize.

@connexallcloud @mloskot @adeturner

[mloskot]

@fady-azmy-msft I'd start with pointing me/us at any current documentation that possibly address the issues I mentioned in my https://github.com/microsoft/Windows-Containers/issues/311#issuecomment-1442436827 I am aware things might have changed since Feb 2023, features and supports might have arrived and improved, but addressing 'old' comments is useful, even if for the sake of the records :)

[adeturner]

@fady-azmy-msft My request was just to monitor metrics of windows containers with the same degree of visibility that we have with Linux containers and cAdvisor. If there is documentation explaining how to do this in non-kubernetes environments please share

[fady-azmy-msft]

Thank you for sharing this feedback. I've assigned the issue to @miremsft who'll be looking at the observability story for windows containers.

[jwilsonCX]

Been a little quiet in this thread lately. Hoping the bot doesn't close this joint down. Any plan of attack on this one, @miremsft ?

[mloskot]

(anti-bot comment)

[jwilsonCX]

Hi @fady-azmy-msft it's been over four months, and it doesn't appear that your assignee @miremsft has accepted the invitation to our Windows observability party:( Any thoughts on assigning a new DJ so we can keep the music playing?

[miremsft]

Hi there, regarding cAdvisor support for Windows Containers, there are currently no plans on our end to do this, but I will update this thread if plans change. For the timebeing, if you would like to track inidividual Windows Container performance and resource usage, it might help to look into what kubelet metrics you can capture to see this https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/winstats/perfcounters.go

The ones that may be particularly helpful for Windows are commit memory and CPU.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @miremsft, please provide an update or close this issue.

[microsoft-github-policy-service[bot]]

This issue has been open for 30 days with no updates. @miremsft, please provide an update or close this issue.

[mloskot]

(anti-bot comment)