`libgomp: Thread creation failed: Resource temporarily unavailable` when running OpenEmbedded builds

[alistair23]

Describe the bug After creating a toolbox and launching a large build process (an OpenEmbedded build) on a 64-core machine eventually the build fails with this error:

libgomp: Thread creation failed: Resource temporarily unavailable

After that the toolbox can't be used to build even smaller tasks as the same error continue to appear.

The machine isn't heavily loaded and the containers ulimit looks like:

ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 1030086
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 500000
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 16384
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 1030086
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

Steps how to reproduce the behaviour

1. toolbox enter yocto
2. source oe-init-build-env
3. export BB_NUMBER_THREADS=24
4. export PARALLEL_MAKE="-j16"
5. bitbake core-image-minimal

Expected behaviour The OpenEmbedded build should complete

Actual behaviour I see lots of libgomp: Thread creation failed: Resource temporarily unavailable failures

Output of toolbox --version (v0.0.90+) toolbox version 0.0.99.2

Toolbox package info (rpm -q toolbox) toolbox-0.0.99.2-1.fc34.x86_64

Output of podman version

Version:      3.2.2
API Version:  3.2.2
Go Version:   go1.16.4
Built:        Sat Jun 26 09:49:19 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman) podman-3.2.2-1.fc34.x86_64

Info about your OS Fedora Silverblue 34

[olivergs]

Hi @alistair23

I've never used this setups, but I've found this:

https://stackoverflow.com/questions/53351194/openmp-libgomp-thread-creation-failed-resource-temporarily-unavailable-when

Seems that as you're running the toolbox as a regular user, the toolbox processes inside the container has the same limits as a regular user.

Maybe tweaking that resource limits using ulimit or just using root to execute the process can fix it.

[debarshiray]

Does a similar build successfully finish on similar hardware without Toolbox?

You could possibly unlock the Silverblue OSTree image temporarily and try to build it directly on the host to test.

[debarshiray]

What does ulimit -a look like on the host?

[alistair23]

A similar build does work on similar hardware when running a different Linux distro. I haven't tested on SilverBlue without toolbox, but I suspect it would work. I can test that though.

real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 1030065
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 81920
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 3030086
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[aboseley]

I'm hitting the same issue running an open-embedded build

libgomp: Thread creation failed: Resource temporarily unavailable

It works when I limit it to 16 threads, but fails at 32. How did you workaround this issue?

[alistair23]

I don't have a workaround unfortunately, I just need to limit the resources used for OE builds

[ric96]

This issue is still persisting. I have enable infinite task limit in user-.slice and set all ulimit to (soft) unlimited. I am still facing this error, along with cpio build error:

|     create archive failed: cpio: write failed - Resource temporarily unavailable