Pixi eats all my memory and then dies

flying-sheep commented 4 days ago

Checks

[x] I have checked that this issue has not already been reported.
[x] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

git clone https://github.com/holoviz/holoviews.git
git checkout 1f534a1b777bd87a8dc3ddbf299a7cae44159540
pixi install

it gets further than that, but it’s always a race against time to be able to copy it:

⠚ updating lock-file   [00:00:03] [━━━━━━━─────────────────────────────────]   14/82                                                                                                                                                                                                                      
  ⠉ test-311:linux-64    [00:00:03] resolving conda
  ⠉ test-311:win-64      [00:00:03] resolving conda
  ⠉ test-311:osx-64      [00:00:03] resolving conda
  ⠉ test-312:linux-64    [00:00:03] resolving conda
  ⠉ test-312:win-64      [00:00:03] resolving conda
  ⠉ test-ui:linux-64     [00:00:03] resolving conda
  ⠉ test-312:osx-64      [00:00:03] resolving conda
  ⠉ test-ui:osx-64       [00:00:03] resolving conda
  ⠉ test-ui:win-64       [00:00:03] resolving conda
  ⠉ test-gpu:linux-64    [00:00:03] resolving conda
  ⠉ test-core:linux-64   [00:00:03] resolving conda
  ⠤ default:linux-64     [00:00:01] resolving conda
  ⠲ default:win-64       [00:00:00] resolving conda
  ⠲ default:osx-64       [00:00:00] resolving conda
  ⠚ build:linux-64       [00:00:00] loading repodata
  ⠙ build:osx-64         [00:00:00] loading repodata

Issue description

I have 16GB of memory, and pixi doesn’t seem to have a setting to reduce the number of threads/processes it uses.

I’m pretty sure the OOM killer kills it.

Expected behavior

not that.

ruben-arts commented 4 days ago

Thanks for reporting!

flying-sheep commented 4 days ago

No problem! Some more details:

I’m on Arch Linux
I did witness the memory climbing until it almost filled up, so the OOM killer hypothesis is is a very informed guess

AvishekMondalQC commented 3 days ago

Implementing a memory limit as suggested here https://github.com/prefix-dev/pixi/issues/2214#issuecomment-2399218804 would be super helpful!

ruben-arts commented 3 days ago

I discussed that with @baszalmstra the other day but it doesn't seem like a memory limit is something we can technically do. But we do need to find a solution for this. Turning off/limiting concurrency should be doable.

prefix-dev / pixi