search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15k stars 4.67k forks source link

The execution time is too long in Kubernetes #69050

Open jerviscui opened 2 years ago

jerviscui commented 2 years ago

Hi @msallin I see your issue Question: Enviornment.ProcessorCount, Kubernetes/OpenShift and ThreadPool Min Threads #41586, how about this question? Did you find out what the problem was?

I used dotnet 6.0 in k8s, and I'm having a similar problem. Application took too long more than expected.

ghost commented 2 years ago

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

Issue Details
Hi @msallin I see your issue [Question: Enviornment.ProcessorCount, Kubernetes/OpenShift and ThreadPool Min Threads #41586](https://github.com/dotnet/runtime/issues/41586), how about this question? Did you find out what the problem was? I used dotnet 6.0 in k8s, and I'm having a similar problem. Application took too long more than expected.
Author: jerviscui
Assignees: -
Labels: `area-System.Threading`, `tenet-performance`, `untriaged`
Milestone: -
deeprobin commented 2 years ago

Is there a good way to profile a .NET App in Kubernetes with collecting the overhead of Kubernetes, Docker, Cgroups, ...?

jerviscui commented 2 years ago

@deeprobin You can try dotnet-monitor or appmetrics.

https://github.com/dotnet/dotnet-monitor https://github.com/AppMetrics/AppMetrics

deeprobin commented 2 years ago

@jerviscui Can you share some system details? OS; libc/musl; CPU; Available memory?

jerviscui commented 2 years ago

I'm running in CentOS 7 and K8s 1.2. I test the API, 1% of 1000 http requests will take too long time. By means of the log in the method, Time spent between method return and middleware.

deeprobin commented 2 years ago

@jerviscui CentOS is a bit special. Do you know if your CentOS uses (g)libc or musl?

And do you see a significant performance difference when running this directly on the host without virtualization?

Could you tell me some specs of the host system of the k8s-node? The count of processors, processor sockets, memory lanes or NUMA nodes could be crucial, since .NET does not yet allocate NUMA-aware so well.

deeprobin commented 2 years ago

@mangod9 As far as I could see from the fabricbot.json you are responsible for the threading. Do you have any suggestions or ideas what the problem could be?

jerviscui commented 2 years ago

The node is a Tencent Cloud VIRTUAL machine. It has 16 core CPU and 32G memory. And I don't know accurate hardware specifications.

OS: CentOS Linux release 7.6.1810 (Core) Kernel: Linux VM-2-19-centos 3.10.0-1160.62.1.el7.x86_64

I've been busy with work lately and haven't tested further. I will test by monitoring system resources and chang the GC Model.

deeprobin commented 2 years ago

Can you install numactl and provide the output of numactl --hardware?

mangod9 commented 2 years ago

@jerviscui could you please provide additional context on what the issue is? Ideally a simple repro for the issue would be helpful in diagnosing further. Thx.