istvan-veeva
commented
4 years ago Hi,
It seems that I was able to sort this out.
I had to increase node.js memory using:
$ export NODE_OPTIONS=--max_old_space_size=8192
Then the transformation was successfully executed:
$ sfdx tmtools:tm1:transform -s output/ ... Command Succeded : tmtools:tm1:transform completed successfully
VivekMChawla
Hello,
I am running $ sfdx tmtools:tm1:transform
command with a fairly significant data volume (5,000+ TM1,0 territories and ~2.5 millions AccountShare records) and I am receiving a Javascript heap out of memory exception:
===================
$ sfdx tmtools:tm1:transform -s output/ ... Transforming TM1 Data/Metadata to TM2... ✔ Prepare Environment for TM1 to TM2 Transformation ⠋ Transform TM1 Metadata to TM2 Metadata → [3s] Transforming TM1 metadata to TM2 metadata Write Transformed TM2 Metadata to the Local Filesystem Write Intermediate TM2 Data to the Local Filesystem
<--- Last few GCs --->
[16247:0x2d227d0] 40451 ms: Mark-sweep 1395.9 (1424.7) -> 1395.4 (1425.2) MB, 1281.1 / 0.0 ms (average mu = 0.127, current mu = 0.003) allocation failure scavenge might not succeed [16247:0x2d227d0] 41761 ms: Mark-sweep 1396.1 (1425.2) -> 1395.6 (1425.7) MB, 1306.4 / 0.0 ms (average mu = 0.068, current mu = 0.003) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x36aa9d99e6e9
1: outputFile [0x1baa4dec7871] [/home/ec2-user/.local/share/sfdx/node_modules/territory-management-toolkit/nodemodules/fs-extra/lib/output/index.js:~9] [pc=0x1c24cc62cd36](this=0x186753a82881 ...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
=========================================
Can you advise me how I can resolve it?
Thanks a lot, Istvan
P.S.:
The entire trace log:
[ec2-user@ip-XXXXX TM3]$ sfdx tmtools:tm1:transform -s output/
---------´ │ v1.0.0 │ ( _´U_ ) ╰──────────────────────────╯ /A\ / | ~ |__'._.'
´
|° ´ YStarting TM1 Transformation Interview...
TM1 Extraction Directory: ? Path to the directory containing tm1-extraction.json? /home/ec2-user/tm2.0/TM3/output
OPTIONS VALUES
TM1 Extraction Directory: /home/ec2-user/tm2.0/TM3/output Alias: undefined
Territories: 5165
User/Territory Assignments: 2424
Assignment Rules: 10
Assignment Rule Items: 10
Account Shares: 2381701
Account Sharing Rules: 0 Criteria-Based | 0 Owner-Based Lead Sharing Rules: 0 Criteria-Based | 0 Owner-Based Opportunity Sharing Rules: 0 Criteria-Based | 0 Owner-Based
Review Your Settings: ? Transform TM1 configuration (data/metadata) to TM2 using the above settings? Yes
Transforming TM1 Data/Metadata to TM2... ✔ Prepare Environment for TM1 to TM2 Transformation ⠋ Transform TM1 Metadata to TM2 Metadata → [3s] Transforming TM1 metadata to TM2 metadata Write Transformed TM2 Metadata to the Local Filesystem Write Intermediate TM2 Data to the Local Filesystem
<--- Last few GCs --->
[16247:0x2d227d0] 40451 ms: Mark-sweep 1395.9 (1424.7) -> 1395.4 (1425.2) MB, 1281.1 / 0.0 ms (average mu = 0.127, current mu = 0.003) allocation failure scavenge might not succeed [16247:0x2d227d0] 41761 ms: Mark-sweep 1396.1 (1425.2) -> 1395.6 (1425.7) MB, 1306.4 / 0.0 ms (average mu = 0.068, current mu = 0.003) allocation failure scavenge might not succeed
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x36aa9d99e6e9
1: outputFile [0x1baa4dec7871] [/home/ec2-user/.local/share/sfdx/node_modules/territory-management-toolkit/nodemodules/fs-extra/lib/output/index.js:~9] [pc=0x1c24cc62cd36](this=0x186753a82881 ...
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x8dc510 node::Abort() [/usr/local/lib/sfdx/bin/node] 2: 0x8dc55c [/usr/local/lib/sfdx/bin/node] 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, bool) [/usr/local/lib/sfdx/bin/node] 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, bool) [/usr/local/lib/sfdx/bin/node] 5: 0xec7bf2 [/usr/local/lib/sfdx/bin/node] 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/lib/sfdx/bin/node] 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/lib/sfdx/bin/node] 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/lib/sfdx/bin/node] 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/lib/sfdx/bin/node] 10: 0xea0466 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [/usr/local/lib/sfdx/bin/node] 11: 0xea0cea v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object, v8::internal::PretenureFlag) [/usr/local/lib/sfdx/bin/node] 12: 0xea12a7 v8::internal::Factory::NewUninitializedFixedArray(int, v8::internal::PretenureFlag) [/usr/local/lib/sfdx/bin/node] 13: 0xe62a20 [/usr/local/lib/sfdx/bin/node] 14: 0xe62cdf [/usr/local/lib/sfdx/bin/node] 15: 0xfd9afe v8::internal::JSObject::AddDataElement(v8::internal::Handle, unsigned int, v8::internal::Handle, v8::internal::PropertyAttributes, v8::internal::ShouldThrow) [/usr/local/lib/sfdx/bin/node]
16: 0xff3262 v8::internal::Object::AddDataProperty(v8::internal::LookupIterator , v8::internal::Handle, v8::internal::PropertyAttributes, v8::internal::ShouldThrow, v8::internal::Object::StoreFromKeyed) [/usr/local/lib/sfdx/bin/node]
17: 0x100dd7d v8::internal::Object::SetProperty(v8::internal::LookupIterator, v8::internal::Handle, v8::internal::LanguageMode, v8::internal::Object::StoreFromKeyed) [/usr/local/lib/sfdx/bin/node]
18: 0x11686c5 v8::internal::Runtime::SetObjectProperty(v8::internal::Isolate , v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::LanguageMode) [/usr/local/lib/sfdx/bin/node]
19: 0x1169820 v8::internal::Runtime_SetProperty(int, v8::internal::Object*, v8::internal::Isolate) [/usr/local/lib/sfdx/bin/node]
20: 0x1c24cc35be1d
/usr/local/bin/sfdx: line 58: 16247 Aborted SFDX_BINPATH="$DIR/$BIN_NAME" "$NODE_PATH" "${NODE_FLAGS[@]}" "$DIR/$MAIN_NAME" "$@"
[ec2-user@ip-XXXXXXX TM3]$