[Bug?]: "Libzip Error: Malloc failure" when installing certain packages

MLoughry commented 11 months ago

Self-service

[X] I'd be willing to implement a fix

Describe the bug

I just moved my team from v1 to v4.02 this past week. Unfortunately, when trying to upgrade one of our Microsoft-internal packages, we ran into the following error:

➤ YN0001: │ Libzip Error: Malloc failure
    at ZipFS.makeLibzipError ([worker eval]:1:296093)
    at ZipFS.getBufferAndClose ([worker eval]:1:296860)
    at ZipFS.saveAndClose ([worker eval]:1:298325)
    at convertToZipWorker ([worker eval]:1:407245)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MessagePort.<anonymous> ([worker eval]:1:409017)

I have traced the line throwing the error, but I'm not familiar enough with the code or libzip to debug further.

This seems to have regressed between 3.7.0 and 4.0.0, because the former installs the package without issues. My suspicion is that it's related to #4247.

To reproduce

Unfortunately, the package is Microsoft internal and cannot be freely shared. I have reached out to the owners of the package to see if we can share it out for this repro (potentially with an NDA), and will update when I have more information.

Environment

System:
    OS: Linux 6.2 Ubuntu 22.04.3 LTS 22.04.3 LTS (Jammy Jellyfish)
    CPU: (32) x64 AMD EPYC 7763 64-Core Processor
  Binaries:
    Node: 18.18.0 - /tmp/xfs-6327a856/node
    Yarn: 4.0.0 - /tmp/xfs-6327a856/yarn
    npm: 9.8.1 - /usr/local/share/nvm/versions/node/v18.18.0/bin/npm

Additional context

No response

arcanis commented 11 months ago

"Malloc failure" would suggest a memory issue. Is it a very large package ?

MLoughry commented 11 months ago

The zip archive is about 4MB, and unzipped it’s about 6MB.

MLoughry commented 11 months ago

Ok, scratch that last comment. Another dev pointed out it could be one of the package’s dependencies, and it turns out that it was: a localized strings package that is 245MB zipped and 1.2GB unzipped.

I'm going to go browbeat whoever thought that was okay, but the fact remains that this is still a regression from 3.7.0 ☹️

merceyz commented 11 months ago

Could you try with compressionLevel: 0 in .yarnrc.yml?

If you're willing could you also test https://github.com/yarnpkg/berry/pull/6011 with compressionLevel set to whatever it was originally?

yarn set version from sources --branch 6011

MLoughry commented 11 months ago

We were already using compressionLevel: 0

compressionLevel: 0

enableGlobalCache: true

nodeLinker: node-modules

npmAlwaysAuth: true

npmRegistryServer: <Internal Azure DevOps Feed>

yarnPath: .yarn/releases/yarn-4.0.2.cjs

Unfortunately, the changes in #6011 did not resolve the issue.

Keisuke69xx commented 10 months ago

I have the same issue. Using compressionLevel:0 and #6011 didn't resolve this issue for us as well.

Archish27 commented 9 months ago

Any updates on this? Please someone help

migueldaipre commented 2 weeks ago

Any way to help here? Same issue using compressionLevel:0 and https://github.com/yarnpkg/berry/pull/6011

yarnpkg / berry