search

Baboo7 / strapi-plugin-import-export-entries

Import/Export data from and to your database in just few clicks.
180 stars 96 forks source link

[BUG] High deepness level with JSON2 runs out of memory even with simple schemas #78

Open minht11 opened 2 years ago

minht11 commented 2 years ago

Describe the bug Exporting while using JSON2 format (other formats work fine), with high deepness level, even with simple schemas runs out of memory and crashes.

To Reproduce Steps to reproduce the behavior:

  1. Install 'strapi-plugin-import-export-entries' plugin
  2. Try exporting data with high deepness level say 20 from simple schema using JSON2 format.
  3. Plugin takes very long time and eventually runs out of memory.
  4. See node stack trace below.

Expected behavior Plugin doesn't crash Strapi.

Additional context If applicable, add screenshots to help explain your problem.

Schema used { "kind": "singleType", "collectionName": "texts", "info": { "singularName": "text", "pluralName": "texts", "displayName": "Text", "description": "" }, "options": { "draftAndPublish": false }, "pluginOptions": { "i18n": { "localized": true } }, "attributes": { "allUseCases": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "orderNow": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "registerNow": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "allLocations": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "learnMore": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "buyNow": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" }, "From": { "pluginOptions": { "i18n": { "localized": true } }, "type": "string" } } }
Terminal logs [2022-11-09 16:43:29.237] http: GET api:api/admin/content-manager/singleType/api::text.text?plugins[i18n][locale]=en (24 ms) 200 [2022-11-09 16:43:29.608] http: GET /admin/project-type (12 ms) 200 [2022-11-09 16:43:29.692] http: POST /admin/renew-token (13 ms) 200 [2022-11-09 16:43:29.698] http: GET /admin/init (3 ms) 200 [2022-11-09 16:43:29.707] http: GET /admin/telemetry-properties (2 ms) 200 [2022-11-09 16:43:29.842] http: GET /admin/users/me/permissions (42 ms) 200 [2022-11-09 16:43:29.852] http: GET /admin/information (54 ms) 200 [2022-11-09 16:43:29.876] http: GET /admin/users/me (72 ms) 200 [2022-11-09 16:43:29.889] http: GET /i18n/locales (9 ms) 200 [2022-11-09 16:43:30.051] http: GET /content-manager/components (6 ms) 200 [2022-11-09 16:43:30.069] http: GET /content-manager/content-types (14 ms) 200 [2022-11-09 16:43:30.085] http: GET /content-manager/content-types-settings (9 ms) 200 [2022-11-09 16:43:30.112] http: GET /content-manager/content-types/api::text.text/configuration (9 ms) 200 [2022-11-09 16:43:30.152] http: GET /content-manager/single-types/api::text.text?locale=en (21 ms) 200 <--- Last few GCs ---> [44426:0x150048000] 92857 ms: Scavenge 2042.7 (2086.1) -> 2040.8 (2086.1) MB, 2.8 / 0.0 ms (average mu = 0.335, current mu = 0.270) allocation failure; [44426:0x150048000] 92863 ms: Scavenge 2042.8 (2086.1) -> 2040.8 (2086.1) MB, 4.4 / 0.0 ms (average mu = 0.335, current mu = 0.270) allocation failure; [44426:0x150048000] 92867 ms: Scavenge 2042.9 (2086.1) -> 2041.0 (2086.3) MB, 3.5 / 0.0 ms (average mu = 0.335, current mu = 0.270) allocation failure; <--- JS stacktrace ---> FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 0x1024f7938 node::Abort() [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 2: 0x1024f7ac0 node::errors::TryCatchScope::~TryCatchScope() [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 3: 0x102649944 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 4: 0x1027f4498 v8::internal::EmbedderStackStateScope::EmbedderStackStateScope(v8::internal::Heap*, v8::internal::EmbedderStackStateScope::Origin, cppgc::EmbedderStackState) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 5: 0x1027f80a0 v8::internal::Heap::CollectSharedGarbage(v8::internal::GarbageCollectionReason) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 6: 0x1027f5094 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*, v8::GCCallbackFlags) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 7: 0x1027f24ec v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 8: 0x1027e7224 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 9: 0x1027e7a54 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 10: 0x1027cd888 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 11: 0x102b5d904 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 12: 0x102ea902c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 13: 0x102e3ab80 Builtins_KeyedStoreIC_Megamorphic [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 14: 0x107dc9288 15: 0x107b0bbf0 16: 0x107b12d24 17: 0x107b131e0 18: 0x107b131e0 19: 0x107b131e0 20: 0x107b131e0 21: 0x107b131e0 22: 0x107b131e0 23: 0x107b131e0 24: 0x107b131e0 25: 0x107b131e0 26: 0x107b131e0 27: 0x107b131e0 28: 0x107b131e0 29: 0x107b131e0 30: 0x107d9b7fc 31: 0x107d9b7fc 32: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 33: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 34: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 35: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 36: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 37: 0x102e34198 Builtins_InterpreterEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 38: 0x107e51fd0 39: 0x107eb15b0 40: 0x107e51fd0 41: 0x107ed9108 42: 0x107e51fd0 43: 0x107eb858c 44: 0x102e65ef4 Builtins_AsyncFunctionAwaitResolveClosure [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 45: 0x102ef46d8 Builtins_PromiseFulfillReactionJob [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 46: 0x102e57c4c Builtins_RunMicrotasks [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 47: 0x102e323a4 Builtins_JSRunMicrotasksEntry [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 48: 0x102775d38 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 49: 0x102776228 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 50: 0x102776404 v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 51: 0x10279ca5c v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 52: 0x10279d2f4 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 53: 0x1026ba17c v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 54: 0x1026b9c78 v8::internal::MaybeHandle v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::Handle, v8::internal::BuiltinArguments) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 55: 0x1026b94a4 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 56: 0x102ea916c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 57: 0x107b748a8 58: 0x102e324d0 Builtins_JSEntryTrampoline [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 59: 0x102e32164 Builtins_JSEntry [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 60: 0x102775d68 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 61: 0x10277529c v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle, v8::internal::Handle, int, v8::internal::Handle*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 62: 0x102665884 v8::Function::Call(v8::Local, v8::Local, int, v8::Local*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 63: 0x102444eec node::InternalCallbackScope::Close() [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 64: 0x10244548c node::InternalMakeCallback(node::Environment*, v8::Local, v8::Local, v8::Local, int, v8::Local*, node::async_context) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 65: 0x10245a310 node::AsyncWrap::MakeCallback(v8::Local, int, v8::Local*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 66: 0x10251257c node::(anonymous namespace)::Parser::OnStreamRead(long, uv_buf_t const&) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 67: 0x1025a80d4 node::LibuvStreamWrap::OnUvRead(long, uv_buf_t const*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 68: 0x102e1f080 uv__stream_io [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 69: 0x102e271b4 uv__io_poll [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 70: 0x102e158cc uv_run [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 71: 0x102445ccc node::SpinEventLoop(node::Environment*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 72: 0x102532060 node::NodeMainInstance::Run(int*, node::Environment*) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 73: 0x102531d38 node::NodeMainInstance::Run() [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 74: 0x1024ca944 node::Start(int, char**) [/Users/justas/.nvm/versions/node/v18.4.0/bin/node] 75: 0x1076690f4
JWesorick commented 2 years ago

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Baboo7 commented 1 year ago

More and more people seems to face performance issue when exporting. Sounds like it's time to optimise 😌 I'll dig into that

Baboo7 commented 1 year ago

https://github.com/Baboo7/strapi-plugin-import-export-entries/issues/77

JWesorick commented 1 year ago

I ran into this one because I didn't know how deep I needed to export so I just picked the highest to see what came out.

sukapyx commented 1 year ago

I ran into this one because I didn't know how deep I needed to export so I just picked the highest to see what came out.

Same here.

kevinvugts commented 6 months ago

@Baboo7 this issue is still present :)