search

miketalbot / js-coroutines

60fps with Javascript Coroutines for idle processing and animation
Other
539 stars 7 forks source link

Nodejs memory error #11

Closed VldMrgnn closed 3 years ago

VldMrgnn commented 3 years ago

Hello

I experience an issue on Nodejs 15.3.0 ( but also on prior version ) express server,

after requiring js-coroutines e.g: ... const mapAsync = require('js-coroutines').mapAsync ...

server crashes after starting in a few minutes.

After not referencing js-coroutines everything goes ok even if I do not uninstall the package.

I would really like to use the library in production. on server as well.

On client side (JS) is working as expected.

thank you,

<--- Last few GCs --->

[23344:00000263D8DCE310] 369156 ms: Mark-sweep 4045.0 (4135.1) -> 4033.4 (4136.9) MB, 3324.5 / 0.1 ms (average mu = 0.129, current mu = 0.029) task scavenge might not succeed [23344:00000263D8DCE310] 373604 ms: Mark-sweep 4046.0 (4136.9) -> 4034.7 (4138.4) MB, 4341.8 / 0.0 ms (average mu = 0.072, current mu = 0.024) task scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 00007FF6751B33EF v8::internal::Isolate::ArchiveSpacePerThread+4447 2: 00007FF67514DEA6 v8::internal::MicrotaskQueue::GetMicrotasksScopeDepth+65030 3: 00007FF67514EDA6 node::OnFatalError+294 4: 00007FF675A49FEE v8::Isolate::ReportExternalAllocationLimitReached+94 5: 00007FF675A2E6FD v8::SharedArrayBuffer::Externalize+781 6: 00007FF6758C814C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468 7: 00007FF6758D3831 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1281 8: 00007FF6758D07F9 v8::internal::Heap::PageFlagsAreConsistent+2377 9: 00007FF6758C5371 v8::internal::Heap::CollectGarbage+1969 10: 00007FF675874A67 v8::internal::ItemParallelJob::Item::TryMarkingAsProcessing+3079 11: 00007FF6750E00AF v8::CFunction::ReturnInfo+17023 12: 00007FF6750DE8D6 v8::CFunction::ReturnInfo+10918 13: 00007FF67520BAEB uv_async_send+331 14: 00007FF67520B27C uv_loop_init+1292 15: 00007FF67520B41A uv_run+202 16: 00007FF6751DAC94 node::SpinEventLoop+308 17: 00007FF675108E07 EVP_CIPHER_CTX_buf_noconst+52551 18: 00007FF67517CF8B node::Start+283 19: 00007FF674FB7BEC RC4_options+344812 20: 00007FF675FD1C3C v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+153276 21: 00007FF946C77C24 BaseThreadInitThunk+20 22: 00007FF947BCD4D1 RtlUserThreadStart+33

miketalbot commented 3 years ago

Interesting, I'll write some tests around using this on a NodeJS server. This is primarily aimed at browser environments but I can see a server use case

miketalbot commented 3 years ago

Verified a server running for one hour and no memory impact by refactoring the polyfill.

VldMrgnn commented 3 years ago

Hello Mike,

I made a new attempt on the same server application and it crashed in 5 minutes.

Then I used similar code on an other server application and there was no error during few hours of running. (same node.js different dependencies )

So, it seems that there is an incompatible library in my setup.

The code I used is trivial and it runs ok.

The package.json of the app where it crashes is kind of messy so I will try again after some cleanup.

Anyway the package is:

{
...
  "dependencies": {
    "@azure/msal-node": "^1.0.0-beta.3",
    "@babel/runtime": "^7.5.5",
    "async": "^3.1.0",
    "body-parser": "^1.19.0",
    "bottleneck": "^2.19.5",
    "bull": "^3.12.1",
    "bull-arena": "^2.7.1",
    "chalk": "^2.4.2",
    "client-certificate-auth": "^0.3.0",
    "compression": "^1.7.4",
    "connect-redis": "^5.0.0",
    "cookie-session": "^1.4.0",
    "cors": "^2.8.5",
    "create-html": "^4.1.0",
    "cron-parser": "^2.13.0",
    "crypto": "^1.0.1",
    "crypto-js": "^3.1.9-1",
    "csurf": "^1.11.0",
    "date-fns": "^2.16.1",
    "deep-diff": "^1.0.2",
    "deep-object-diff": "^1.1.0",
    "dotenv": "^8.2.0",
    "eslint-import-resolver-babel-module": "^5.2.0",
    "eslint-plugin-security-node": "^1.0.14",
    "express": "^4.17.1",
    "express-jwt": "^6.0.0",
    "express-jwt-authz": "^2.4.1",
    "express-openid-connect": "^2.0.0",
    "express-pino-logger": "^4.0.0",
    "express-promise-router": "^4.0.1",
    "express-rate-limit": "^5.1.3",
    "express-redis-cache": "^1.1.3",
    "express-session": "^1.17.1",
    "express-validator": "^6.6.1",
    "faker": "^5.1.0",
    "fast_array_intersect": "^1.1.0",
    "fast-array-diff": "^1.0.0",
    "fetch-cookie": "^0.7.3",
    "fs": "^0.0.1-security",
    "handlebars": "^4.7.6",
    "handy-redis": "^2.1.0",
    "helmet": "^4.2.0",
    "hpp": "^0.2.3",
    "http-errors": "^1.7.2",
    "https": "^1.0.0",
    "immutability-helper": "^3.0.1",
    "iso-8859-2": "^1.0.0",
    "isomorphic-fetch": "^3.0.0",
    "js-coroutines": "^2.4.10",
    "jslint": "^0.12.1",
    "json-parse-safe": "^2.0.0",
    "jsonschema": "^1.4.0",
    "jsonwebtoken": "^8.5.1",
    "jwks-rsa": "^1.12.2",
    "lodash": "^4.17.20",
    "lodash-joins": "^2.0.3",
    "lodash.product": "^18.9.19",
    "make-runnable": "^1.3.8",
    "md5": "^2.2.1",
    "memorystore": "^1.6.2",
    "moment": "^2.29.1",
    "moment-timezone": "^0.5.31",
    "nanoid": "^2.1.7",
    "node-fetch": "^2.6.1",
    "node-fetch-cookies": "^1.3.4",
    "node-firebird": "^0.8.9",
    "node-redis-scan": "^1.1.1",
    "nodemailer": "^6.3.0",
    "nodemailer-smtp-transport": "^2.7.4",
    "nodemon": "^2.0.6",
    "npm": "^7.6.3",
    "path": "^0.12.7",
    "pino": "^5.12.6",
    "pino-pretty": "^3.2.0",
    "prettier": "^2.1.2",
    "redis": "^2.8.0",
    "redis-async-gen": "^0.1.3",
    "redis-cache-express-middleware": "^1.0.4",
    "scramjet": "^4.35.10",
    "selfsigned": "^1.10.8",
    "simple-continuation-local-storage": "^1.0.22",
    "uuid": "^8.3.1"
  },
  "devDependencies": {
    "@babel/core": "^7.8.6",
    "@babel/plugin-proposal-optional-chaining": "^7.2.0",
    "@babel/plugin-transform-runtime": "^7.5.5",
    "@babel/preset-env": "^7.8.6",
    "@babel/register": "^7.8.6",
    "eslint": "^7.16.0",
    "eslint-config-airbnb-base": "^14.2.1",
    "eslint-config-standard": "^16.0.2",
    "eslint-plugin-babel": "^5.3.0",
    "eslint-plugin-import": "^2.22.1",
    "eslint-plugin-node": "^11.1.0",
    "eslint-plugin-promise": "^4.2.1",
    "eslint-plugin-react": "^7.21.5"
  },
 ...
}

The error :

<--- Last few GCs --->

[7548:000002321B42C200]   354229 ms: Scavenge 4044.0 (4124.9) -> 4039.2 (4126.1) MB, 4.4 / 0.0 ms  (average mu = 0.587, current mu = 0.548) task
[7548:000002321B42C200]   354287 ms: Scavenge 4045.5 (4126.4) -> 4040.7 (4143.9) MB, 5.0 / 0.0 ms  (average mu = 0.587, current mu = 0.548) task
[7548:000002321B42C200]   357514 ms: Mark-sweep 4046.1 (4143.9) -> 4028.7 (4143.9) MB, 3182.7 / 0.0 ms  (average mu = 0.455, current mu = 0.209) allocation failure GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
 1: 00007FF653FE33EF v8::internal::Isolate::ArchiveSpacePerThread+4447
 2: 00007FF653F7DEA6 v8::internal::MicrotaskQueue::GetMicrotasksScopeDepth+65030
 3: 00007FF653F7EDA6 node::OnFatalError+294
 4: 00007FF654879FEE v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF65485E6FD v8::SharedArrayBuffer::Externalize+781
 6: 00007FF6546F814C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1468
 7: 00007FF6546E3068 v8::internal::NativeContextInferrer::Infer+59352
 8: 00007FF6546C7C3F v8::internal::MarkingWorklists::Local::SwitchToContextSlow+60639
 9: 00007FF6546DBFE1 v8::internal::NativeContextInferrer::Infer+30545
10: 00007FF6546D313D v8::internal::MarkCompactCollector::EnsureSweepingCompleted+6285
11: 00007FF6546DB10E v8::internal::NativeContextInferrer::Infer+26750
12: 00007FF6546DF3CB v8::internal::NativeContextInferrer::Infer+43835
13: 00007FF6546E92B2 v8::internal::ItemParallelJob::Task::RunInternal+18
14: 00007FF6546E9241 v8::internal::ItemParallelJob::Run+641
15: 00007FF6546BB3ED v8::internal::MarkingWorklists::Local::SwitchToContextSlow+9357
16: 00007FF6546D36DE v8::internal::MarkCompactCollector::EnsureSweepingCompleted+7726
17: 00007FF6546D1E24 v8::internal::MarkCompactCollector::EnsureSweepingCompleted+1396
18: 00007FF6546CF528 v8::internal::MarkingWorklists::Local::SwitchToContextSlow+91592
19: 00007FF6546FDF91 v8::internal::Heap::LeftTrimFixedArray+945
20: 00007FF654700175 v8::internal::Heap::PageFlagsAreConsistent+709
21: 00007FF6546F5371 v8::internal::Heap::CollectGarbage+1969
22: 00007FF6546A4A67 v8::internal::ItemParallelJob::Item::TryMarkingAsProcessing+3079
23: 00007FF653F100AF v8::CFunction::ReturnInfo+17023
24: 00007FF653F0E8D6 v8::CFunction::ReturnInfo+10918
25: 00007FF65403BAEB uv_async_send+331
26: 00007FF65403B27C uv_loop_init+1292
27: 00007FF65403B41A uv_run+202
28: 00007FF65400AC94 node::SpinEventLoop+308
29: 00007FF653F38E07 EVP_CIPHER_CTX_buf_noconst+52551
30: 00007FF653FACF8B node::Start+283
31: 00007FF653DE7BEC RC4_options+344812
32: 00007FF654E01C3C v8::internal::compiler::RepresentationChanger::Uint32OverflowOperatorFor+153276
33: 00007FFD17627034 BaseThreadInitThunk+20
34: 00007FFD17762651 RtlUserThreadStart+33
[nodemon] app crashed - waiting for file changes before starting...