search

antonmedv / fx

Terminal JSON viewer & processor
https://fx.wtf
MIT License
19.08k stars 438 forks source link

Maximum file size? #165

Closed ghostsquad closed 2 years ago

ghostsquad commented 3 years ago

I have a 500MB json file that I need to analyze, but it looks like that might be too large. Is there anything that can be done about this?

<--- Last few GCs --->

[9339:0x10281a000]    13436 ms: Mark-sweep 2047.2 (2049.9) -> 2047.2 (2051.1) MB, 4.4 / 0.0 ms  (+ 613.2 ms in 34 steps since start of marking, biggest step 41.6 ms, walltime since start of marking 848 ms) (average mu = 0.294, current mu = 0.273) finalize

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1009288f9]
    1: StubFrame [pc: 0x1009297d3]
Security context: 0x39c340ec0919 <JSObject>
    2: E(aka E) [0x39c35e1e2391] [/usr/local/Cellar/fx/20.0.2/libexec/lib/node_modules/fx/node_modules/lossless-json/dist/lossless-json.js:~1] [pc=0x3080f804d0f](this=0x39c3fef804d1 <undefined>)
    3: /* anonymous */(aka /* anonymous */) [0x39c342f80159] [/usr/local/Cellar/fx/20.0.2/libexec/lib/node_modules/fx/node_modules/lo...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
brainstorm commented 2 years ago

Similar issue here with a 400MB one:

% fx H-UDI.json

<--- Last few GCs --->

[97379:0x140008000]    38783 ms: Scavenge 4061.1 (4123.1) -> 4057.3 (4124.1) MB, 7.5 / 0.0 ms  (average mu = 0.293, current mu = 0.089) allocation failure
[97379:0x140008000]    38793 ms: Scavenge 4062.1 (4124.1) -> 4059.0 (4126.4) MB, 5.8 / 0.0 ms  (average mu = 0.293, current mu = 0.089) allocation failure
[97379:0x140008000]    39542 ms: Scavenge 4064.4 (4126.4) -> 4061.0 (4144.4) MB, 743.4 / 0.0 ms  (average mu = 0.293, current mu = 0.089) allocation failure

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x1027b07a0 node::Abort() [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 2: 0x1027b17a0 node::OnFatalError(char const*, char const*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 3: 0x102908920 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 4: 0x1029088b4 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 5: 0x102a586b8 v8::internal::Heap::EnsureFillerObjectAtTop() [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 6: 0x102a576b8 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 7: 0x102a640a4 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 8: 0x102a64124 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
 9: 0x102a3bbbc v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
10: 0x102cf0e54 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
11: 0x1026262ec Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/opt/homebrew/Cellar/node/17.3.0/bin/node]
12: 0x1025b83cc Builtins_FastNewObject [/opt/homebrew/Cellar/node/17.3.0/bin/node]
13: 0x1025b7d58 Builtins_JSConstructStubGeneric [/opt/homebrew/Cellar/node/17.3.0/bin/node]
14: 0x108f2d1bc
15: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
16: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
17: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
18: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
19: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
20: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
21: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
22: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
23: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
24: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
25: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
26: 0x1025bae98 Builtins_InterpreterEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
27: 0x1025b8cac Builtins_JSEntryTrampoline [/opt/homebrew/Cellar/node/17.3.0/bin/node]
28: 0x1025b8944 Builtins_JSEntry [/opt/homebrew/Cellar/node/17.3.0/bin/node]
29: 0x1029f4b84 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
30: 0x1029f4184 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
31: 0x10291f6f0 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
32: 0x10277a77c node::ExecuteBootstrapper(node::Environment*, char const*, std::__1::vector<v8::Local<v8::String>, std::__1::allocator<v8::Local<v8::String> > >*, std::__1::vector<v8::Local<v8::Value>, std::__1::allocator<v8::Local<v8::Value> > >*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
33: 0x10277b954 node::StartExecution(node::Environment*, char const*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
34: 0x10277b824 node::StartExecution(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
35: 0x1026eebfc node::LoadEnvironment(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
36: 0x1027ee0bc node::NodeMainInstance::Run(int*, node::Environment*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
37: 0x1027edd40 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
38: 0x10277e464 node::Start(int, char**) [/opt/homebrew/Cellar/node/17.3.0/bin/node]
39: 0x104a850f4
zsh: abort      fx H-UDI.json

% ls -alh H-UDI.json
-rw-r--r-- 1 rvalls staff 399M Jan 20 13:20 H-UDI.json
antonmedv commented 2 years ago

https://github.com/antonmedv/fx/blob/master/DOCS.md#memory-usage

antonmedv commented 2 years ago

I rewrote the entire fx in golang. So I belove this issue is fixed.