pixi global upgrade-all fails with "Too many open files (os error 24)" on macOS aarch64

[matthewfeickert]

Checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

This issue is being opened upon request from https://github.com/prefix-dev/pixi/issues/1551#issuecomment-2325612275 given that the examples there don't show further information with the --verbose flag. This issue so far has only been seen on Apple silcon macOS, as far as I know.

% ulimit -n
256
% pixi self-update
✔ pixi is already up-to-date (version 0.29.0)
% pixi global upgrade-all
  × failed to parse '/Users/feickert/.pixi/envs/xrootd/conda-meta/jinja2-3.1.4-pyhd8ed1ab_0.json'
  ╰─▶ Too many open files (os error 24)

% pixi global --verbose upgrade-all
  × failed to parse '/Users/feickert/.pixi/envs/ninja/conda-meta/libev-4.33-h93a5062_2.json'
  ╰─▶ Too many open files (os error 24)

% pixi global --verbose upgrade-all
  × failed to parse '/Users/feickert/.pixi/envs/shellcheck/conda-meta/hist-base-2.8.0-pyhd8ed1ab_1.json'
  ╰─▶ Too many open files (os error 24)

% ulimit -n 2048
% pixi global upgrade-all
Nothing to upgrade

Issue description

For Apple silicon macOS, from time to time (but not always) running pixi global upgrade-all will fail with the error Too many open files (os error 24). If the ulimit maximum open file descriptors is is manually increased this issue isn't observed.

Expected behavior

Have pixi global commands not require manual user changes to ulimit.

[stanmart]

It also happens with pixi exec if the number of dependencies is sufficiently large. For example, the following command runs into this error quite consistently:

pixi exec -s glum -s skglm -s jupyter -s libsvmdata -s matplotlib jupyter lab

[wolfv]

@baszalmstra can we reduce the limit on the number of open files by some more?