dotnet / extensions

This repository contains a suite of libraries that provide facilities commonly needed when creating production-ready applications.
MIT License
2.59k stars 743 forks source link

LinuxUtilizationProvider can't handle 'max' in /sys/fs/cgroup/memory.max #5241

Closed morti12 closed 2 months ago

morti12 commented 3 months ago

Description

We are using the newest OPC Publisher docker image which introduced in version 2.9.8 the newest version of the .NET Extensions 8.6.0. It actually should include a bug fix for #5161 from the version 8.5.0.

On the startup we get the following exception because the /sys/fs/cgroup/memory.max contains max as value which can't be parsed as integer.

When checking it over the console, cat /sys/fs/cgroup/memory.max returns max from this path.

The default value of the /sys/fs/cgroup/memory.max can actually be max. Please have a look into the documentation.

  memory.max
    A read-write single value file which exists on non-root
    cgroups.  The default is "max".

Reproduction Steps

I tried it with different distributions but all of them have a different structure of the cgroup folder. Sry, I'm not a Linux expert...

So it's only not working on this RHEL machine. At least on every start of the docker container we get this message.

It's really hard to reproduce.

Expected behavior

Actual behavior

Because of the max in the memory.max we get the following error:

Unhandled exception. Autofac.Core.DependencyResolutionException: An exception was thrown while activating Azure.IIoT.OpcUa.Publisher.Services.PublisherDiagnosticCollector -> λ:Microsoft.Extensions.Diagnostics.ResourceMonitoring.IResourceMonitor -> Microsoft.Extensions.Diagnostics.ResourceMonitoring.ResourceMonitorService -> Microsoft.Extensions.Diagnostics.ResourceMonitoring.Linux.LinuxUtilizationProvider.
---> Autofac.Core.DependencyResolutionException: An exception was thrown while invoking the constructor 'Void .ctor(Microsoft.Extensions.Options.IOptions`1[Microsoft.Extensions.Diagnostics.ResourceMonitoring.ResourceMonitoringOptions], Microsoft.Extensions.Diagnostics.ResourceMonitoring.Linux.ILinuxUtilizationParser, System.Diagnostics.Metrics.IMeterFactory, System.TimeProvider)' on type 'LinuxUtilizationProvider'.
---> System.InvalidOperationException: Could not parse '/sys/fs/cgroup/memory.max' content. Expected to find available memory in bytes but got 'max' instead.

It looks like that we still have the same problem as it was already solved in the issue #5161

Regression?

No response

Known Workarounds

It's possible to set in docker a memory limit with the argument --memory=2g. Then The library can parse the value.

Then we get another exception for the cpu.max: System.InvalidOperationException: Could not parse '/sys/fs/cgroup/cpu.max'. Expected an integer but got: 'max 100000'.

It seems that the max in the cpu.max always contains the max <number>

  cpu.max
    A read-write two value file which exists on non-root cgroups.
    The default is "max 100000".

    The maximum bandwidth limit.  It's in the following format::

      $MAX $PERIOD

    which indicates that the group may consume upto $MAX in each
    $PERIOD duration.  "max" for $MAX indicates no limit.  If only
    one number is written, $MAX is updated.

Configuration

Host

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.3 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.3 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="[https://www.redhat.com/"](https://www.redhat.com/%22)
DOCUMENTATION_URL="[https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9%22)
BUG_REPORT_URL="[https://bugzilla.redhat.com/"](https://bugzilla.redhat.com/%22)

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"
$ uname -a
Linux sha-ld-2876 5.14.0-362.24.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 15 07:18:13 EST 2024 x86_64 x86_64 x86_64 GNU/Linux

Docker / moby

Docker version 26.1.4-1

Container Image / Base Image

OPC Publisher: "mcr.microsoft.com/iotedge/opc-publisher:2.9.8" which is using the newest Version 8.6.0 of the extension

Base image of the OPC Publisher:

"org.opencontainers.image.base.name": "mcr.microsoft.com/dotnet/aspnet:8.0-cbl-mariner-distroless-amd64"
"org.opencontainers.image.created": "06/13/2024 20:29:13",

Other information

No response