rpm on centos 7 calls fcntl on every FD up to the max in order to set CLOEXEC, and the maximum number of open FDs in docker on our runners was 2**30 - 8 == 1073741816.
(See https://github.com/rpm-software-management/rpm/pull/444).
The result was that builds using centos 7 were taking almost 4 hours instead of a couple minutes.
(We upgraded the version of the thing we were using to run the jobs on Nov 6, and I suspect that caused the ulimit to change from about a million (wasteful, but not a diaster) to about a billion (much worse).)
ulimit can't be configured in a Dockerfile, and there doesn't seem to be a way to pass argument to docker build if you are using the default github action, so just call ulimit before the big yum install.
There might be a way to configure it on our infra runner side too, I don't know.
rpm on centos 7 calls fcntl on every FD up to the max in order to set CLOEXEC, and the maximum number of open FDs in docker on our runners was 2**30 - 8 == 1073741816. (See https://github.com/rpm-software-management/rpm/pull/444).
The result was that builds using centos 7 were taking almost 4 hours instead of a couple minutes. (We upgraded the version of the thing we were using to run the jobs on Nov 6, and I suspect that caused the ulimit to change from about a million (wasteful, but not a diaster) to about a billion (much worse).)
ulimit can't be configured in a Dockerfile, and there doesn't seem to be a way to pass argument to
docker buildif you are using the default github action, so just call ulimit before the big yum install.There might be a way to configure it on our infra runner side too, I don't know.