[Bug]: Too many open files

[philvv]

Bug report

We have been using DD for years with no issue and in the last while, through upgrading our application to php 8.3, we reinstalled datadog.

Since then our servers are crashing due to 'too many open files'. I would assume the version running before was less than 1.4 but I have no record, as the server was rebuilt.

I rebooted all servers through the night, as our application was failing. They have been running for 8 hours. See output from commands on 1 server running ddtrace:

cat /proc/sys/fs/file-nr 582840 0 2097152

sudo lsof | cut -d " " -f 1 | uniq -c | sort -nr | head -n 10 406259 php8.3 225341 dd-ipc-he 175165 php 13272 php-fpm8. 3648 agent 2268 php8.3 1899 php-fpm8. 1725 nginx 1422 php-fpm8. 1408 trace-age

I have disabled ddtrace on 1 server and open files is holding under 10,000.

cat /proc/sys/fs/file-nr 4680 0 2097152

sudo lsof | cut -d " " -f 1 | uniq -c | sort -nr | head -n 10 18482 php-fpm8. 3474 nginx 2500 agent 1803 php-fpm8. 1212 php8.3 753 php-fpm8. 600 php-fpm8. 594 snapd 552 process-a 525 trace-age

I can only assume this is being cause by ddtrace?

PHP version

8.3.10

Tracer or profiler version

1.4.2

Installed extensions

[PHP Modules] bcmath bz2 calendar Core ctype curl date ddappsec ddtrace dom exif FFI fileinfo filter ftp gd gettext gmp hash iconv igbinary intl json libxml mbstring mysqli mysqlnd openssl pcntl pcre PDO pdo_mysql Phar posix random readline redis Reflection session shmop SimpleXML sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zip zlib

[Zend Modules] Zend OPcache ddappsec ddtrace

Output of phpinfo()

No response

Upgrading from

No response

[bwoebi]

Thanks for the report @philvv,

I haven't been observing that yet with ddtrace. Are you able to provide information on what files / sockets are being held open? Is the memory usage of dd-ipc-helper also growing linearly with the number of files?

I'm seeing 406259 php8.3 here - I assume these are PHP CLI processes. Is that a single process? Many CLI processes which never end despite that they could?

[philvv]

Hi,

Are you able to provide information on what files / sockets are being held open? You will need to point to what you need me to show, as I am no server expert.

Is the memory usage of dd-ipc-helper also growing linearly with the number of files? Memory use is growing linearly overtime until it starts to fail. I then reboot, it troughs and grows again linearly. See pic:

Memory use, used to be stable at about 6 gig but now getting over 20 gig (application use is less than August, so its not user related). 25th August was when we recreated the servers running php 8.3 and installed whatever version of dd would have been latest at that time. See pic:

I assume these are PHP CLI processes Its a laravel application running horizon, so cli would be horizon workers and cron jobs. We run about 5 crons per minute. The queue is not heavily used, so typically 2 processes in waiting state per server.

[bwoebi]

@philvv Hm, is it possible for you to join https://chat.datadoghq.com/ and contact me there (you'll find me easily)? For better communication? We've tried to reproduce such an issue already multiple times, but haven't managed to do so ourselves.

The ipc-helper process is definitely not supposed to use gigabytes of memory. We're in fact restarting the sidecar automatically if it uses too much memory, but I suppose memory used by file descriptions might not be included in the self-reporting of helper memory.

[philvv]

Hi @bwoebi,

It maybe worth mentioning that our application has a consistent throughput of 1,400 requests per second, which is spread across 6 servers, so roughly 230 rps per server.

Ye, I will try and join your channel tomorrow.