search

forcedotcom / cli

Salesforce CLI
https://developer.salesforce.com/docs/atlas.en-us.sfdx_cli_reference.meta/sfdx_cli_reference/
BSD 3-Clause "New" or "Revised" License
474 stars 77 forks source link

Converting source in large projects results in JavaScript heap out of memory error #2939

Open jstvz opened 1 week ago

jstvz commented 1 week ago

Summary

Converting source in large projects results in JavaScript heap out of memory error

Steps To Reproduce

  1. git clone --depth 1 https://github.com/SalesforceFoundation/NPSP.git npsp; cd npsp
  2. sf project convert source

Expected result

$ sf project convert source
Source was successfully converted to Metadata API format and written to the location: /private/tmp/npsp/metadataPackage_1719260773403

Actual result

$ sf project convert source

<--- Last few GCs --->

[88567:0x7fc918008000]    99781 ms: Scavenge 3326.0 (4122.4) -> 3313.1 (4124.2) MB, 3.61 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;
[88567:0x7fc918008000]    99807 ms: Scavenge 3329.3 (4125.7) -> 3316.1 (4126.7) MB, 6.89 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;
[88567:0x7fc918008000]    99834 ms: Scavenge 3331.7 (4126.7) -> 3318.7 (4127.7) MB, 4.14 / 0.00 ms  (average mu = 0.281, current mu = 0.226) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x1042f9625 node::Abort() [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 2: 0x1042f980d node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 3: 0x1044a9d1c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 4: 0x104699cf7 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 5: 0x10469e2c7 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 6: 0x10469a615 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 7: 0x1046980b0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 8: 0x10468cdc1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
 9: 0x10468d805 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
10: 0x10466f962 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
11: 0x104aed850 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]
12: 0x104ebf8b6 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/jestevez/.volta/tools/image/node/20.11.0/bin/node]

System Information

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.46.6",
  "nodeVersion": "node-v20.11.0",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/Users/jestevez/.volta/tools/image/packages/@salesforce/cli/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.1.2 (core)",
    "@oclif/plugin-commands 4.0.2 (core)",
    "@oclif/plugin-help 6.2.1 (core)",
    "@oclif/plugin-not-found 3.2.3 (core)",
    "@oclif/plugin-plugins 5.2.4 (core)",
    "@oclif/plugin-search 1.1.2 (core)",
    "@oclif/plugin-update 4.3.5 (core)",
    "@oclif/plugin-version 2.2.3 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.5 (core)",
    "@oclif/plugin-which 3.2.2 (core)",
    "@salesforce/cli 2.46.6 (core)",
    "apex 3.1.18 (core)",
    "auth 3.6.21 (core)",
    "data 3.4.6 (core)",
    "deploy-retrieve 3.9.4 (core)",
    "info 3.3.6 (core)",
    "limits 3.3.12 (core)",
    "marketplace 1.2.12 (core)",
    "org 4.2.1 (core)",
    "packaging 2.4.6 (core)",
    "schema 3.3.11 (core)",
    "settings 2.3.1 (core)",
    "sobject 1.4.11 (core)",
    "source 3.4.2 (core)",
    "telemetry 3.4.0 (core)",
    "templates 56.2.11 (core)",
    "trust 3.7.5 (core)",
    "user 3.5.13 (core)"
  ]
}

Additional information

Attempted workaround from forcedotcom/salesforcedx-vscode#5589, but I still got heap errors on apex 3.1.3 (user) and apex 3.2.0 (user). However installing @salesforce/cli@2.36.8 worked:

{
  "architecture": "darwin-x64",
  "cliVersion": "@salesforce/cli/2.36.8",
  "nodeVersion": "node-v20.11.0",
  "osVersion": "Darwin 22.6.0",
  "rootPath": "/Users/jestevez/.volta/tools/image/packages/@salesforce/cli/lib/node_modules/@salesforce/cli",
  "shell": "zsh",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.13 (core)",
    "@oclif/plugin-commands 3.2.2 (core)",
    "@oclif/plugin-help 6.0.20 (core)",
    "@oclif/plugin-not-found 3.1.1 (core)",
    "@oclif/plugin-plugins 5.0.5 (core)",
    "@oclif/plugin-search 1.0.20 (core)",
    "@oclif/plugin-update 4.2.3 (core)",
    "@oclif/plugin-version 2.0.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.15 (core)",
    "@oclif/plugin-which 3.1.7 (core)",
    "@salesforce/cli 2.36.8 (core)",
    "apex 3.2.0 (user)",
    "auth 3.5.1 (core)",
    "custom-metadata 2.2.13 (user)",
    "data 3.2.4 (core)",
    "deploy-retrieve 3.5.3 (core)",
    "dev 1.1.2 (user)",
    "env 2.1.18 (user)",
    "info 3.1.2 (core)",
    "limits 3.2.1 (core)",
    "marketplace 1.1.0 (core)",
    "org 3.6.3 (core)",
    "packaging 1.21.1 (user)",
    "schema 3.2.0 (core)",
    "settings 2.1.2 (core)",
    "sobject 1.2.1 (core)",
    "source 3.2.2 (core)",
    "telemetry 3.1.17 (core)",
    "templates 56.1.0 (core)",
    "trust 3.5.1 (core)",
    "user 3.4.3 (core)"
  ]
}
github-actions[bot] commented 1 week ago

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

shetzel commented 1 week ago

Thanks for the super simple repro! It's definitely a problem. We'll have to track down where this is coming from. As a temporary workaround you can export NODE_OPTIONS="--max-old-space-size=8192" and that was enough memory to complete the conversion. Alternatively, you can find a version of the @salesforce/plugin-deploy-retrieve plugin that accomplishes the conversion without bumping the heap size.

git2gus[bot] commented 1 week ago

This issue has been linked to a new work item: W-16096068

shetzel commented 1 week ago

@jstvz - I found that sf plugins install deploy-retrieve@3.6.9 or any version previous results in a successful conversion. Any version of the plugin after that results in OOM errors. Until the bug is fixed, you can either bump the heap size or install deploy-retrieve@3.6.9.