search

gund / eslint-plugin-deprecation

ESLint rule that reports usage of deprecated code
GNU Lesser General Public License v3.0
330 stars 39 forks source link

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #67

Open mrrajamanickam-coveo opened 1 year ago

mrrajamanickam-coveo commented 1 year ago

Running this plugin results in out of memory error

<--- Last few GCs --->

[93771:0x110040000]    29282 ms: Scavenge 4043.7 (4125.2) -> 4040.7 (4126.4) MB, 13.2 / 0.0 ms  (average mu = 0.870, current mu = 0.542) allocation failure;
[93771:0x110040000]    29312 ms: Scavenge 4044.6 (4126.4) -> 4042.6 (4130.2) MB, 10.7 / 0.0 ms  (average mu = 0.870, current mu = 0.542) allocation failure;
[93771:0x110040000]    29651 ms: Scavenge 4048.6 (4130.6) -> 4046.6 (4148.1) MB, 316.4 / 0.0 ms  (average mu = 0.870, current mu = 0.542) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
gund commented 1 year ago

This is not very useful I think we need more info to understand this issue. Did you disable all other eslint plugins/rules, whats the project size, what OS and what's the process limit on it.

The simplest advise I can give you is to extend the limits for your node process on the OS level, you can use google or chatgpt to find info how to do that in your case.

Also if it still will be failing I think you should try to run eslint in debug/verbose mode so maybe we will see some more usefull info before it crashes as those logs are practically useless apart from seeing that your node process limit is 4GB (which is low btw).

mrrajamanickam-coveo commented 1 year ago

Sorry, from the info in issue template didn't know if the issue will be looked at - hence didn't bother with much info. Thanks for looking into this. Here is more info Yes, I did try reducing the eslint config to disable all other plugins essentially reducing it to

{
  "ignorePatterns": [
    "node_modules",
    "stencil-generated",
    "dist",
    "www",
    "!.storybook"
  ],
  "env": {
    "jest": true,
    "es6": true
  },
  "parserOptions": {
    "ecmaVersion": 2020,
    "sourceType": "module"
  },
  "overrides": [
    {
      "files": ["**/*.ts", "**/*.tsx"],
      "parser": "@typescript-eslint/parser",
      "plugins": ["@typescript-eslint", "deprecation"],
      "extends": ["./node_modules/gts"],
      "parserOptions": {
        "jsxPragma": "h",
        "project": "./tsconfig.json"
      },
      "rules": {
        "deprecation/deprecation": "warn"
      }
    }
  ],
  "root": true
}

Increasing heap size to NODE_OPTIONS=--max-old-space-size=4096 did not help. But increasing it to 8192 resulted in the plugin running successfully. But this might not work because

My branch: https://github.com/coveo/ui-kit/tree/KIT-2592_eslint_plugin_deprecation Looks like this issue might be related to https://github.com/typescript-eslint/typescript-eslint/issues/1192

Running eslint with --debug flag:

 eslint:file-enumerator Enter the directory: ui-kit/packages/atomic/cypress/e2e +0ms
  eslintrc:cascading-config-array-factory Load config files for ui-kit/packages/atomic/cypress/e2e. +8ms
  eslintrc:cascading-config-array-factory No cache found: ui-kit/packages/atomic/cypress/e2e. +0ms
  eslintrc:config-array-factory Config file not found on ui-kit/packages/atomic/cypress/e2e +8ms
  eslintrc:cascading-config-array-factory Cache hit: ui-kit/packages/atomic/cypress. +0ms
  eslintrc:ignore-pattern Create with: [ IgnorePattern { patterns: [ '/**/node_modules/*' ], basePath: 'ui-kit', loose: false }, IgnorePattern { patterns: [ 'node_modules', 'stencil-generated', 'dist', 'www', '!.storybook' ], basePath: 'ui-kit', loose: false }, IgnorePattern { patterns: [ 'src/external-builds/**/*', 'dist/**/*', 'www/**/*', 'loader/**/*', 'docs/**/*' ], basePath: 'ui-kit/packages/atomic', loose: false } ] +0ms
  eslintrc:ignore-pattern   processed: { basePath: 'ui-kit', patterns: [ '/**/node_modules/*', 'node_modules', 'stencil-generated', 'dist', 'www', '!.storybook', '/packages/atomic/**/src/external-builds/**/*', '/packages/atomic/**/dist/**/*', '/packages/atomic/**/www/**/*', '/packages/atomic/**/loader/**/*', '/packages/atomic/**/docs/**/*' ] } +1ms
  eslintrc:ignore-pattern Check {
  filePath: 'ui-kit/packages/atomic/cypress/e2e/aria-live-selectors.ts',
  dot: false,
  relativePath: 'packages/atomic/cypress/e2e/aria-live-selectors.ts',
  result: false
} +0ms
  eslint:file-enumerator Yield: aria-live-selectors.ts +1ms
  eslintrc:cascading-config-array-factory Load config files for ui-kit/packages/atomic/cypress/e2e. +1ms
  eslintrc:cascading-config-array-factory Cache hit: ui-kit/packages/atomic/cypress/e2e. +0ms
  eslint:cli-engine Lint ui-kit/packages/atomic/cypress/e2e/aria-live-selectors.ts +270ms
  eslint:linter Linting code for ui-kit/packages/atomic/cypress/e2e/aria-live-selectors.ts (pass 1) +257ms
  eslint:linter Verify +0ms
  eslint:linter With ConfigArray: ui-kit/packages/atomic/cypress/e2e/aria-live-selectors.ts +0ms
  eslint:linter Parsing: ui-kit/packages/atomic/cypress/e2e/aria-live-selectors.ts +1ms

<--- Last few GCs --->

[6764:0x130078000]    29528 ms: Scavenge 4044.0 (4124.3) -> 4041.5 (4125.3) MB, 14.4 / 0.0 ms  (average mu = 0.878, current mu = 0.535) allocation failure;
[6764:0x130078000]    29566 ms: Scavenge 4044.8 (4125.3) -> 4043.1 (4129.5) MB, 11.9 / 0.0 ms  (average mu = 0.878, current mu = 0.535) allocation failure;
[6764:0x130078000]    29838 ms: Scavenge 4048.9 (4129.5) -> 4045.5 (4146.8) MB, 255.6 / 0.0 ms  (average mu = 0.878, current mu = 0.535) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100ee94fc node::Abort() [.nvm/versions/node/v18.16.0/bin/node]
 2: 0x100ee96ec node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [.nvm/versions/node/v18.16.0/bin/node]
 3: 0x10103f0fc v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [.nvm/versions/node/v18.16.0/bin/node]
 4: 0x1011ea088 v8::internal::EmbedderStackStateScope::EmbedderStackStateScope(v8::internal::Heap*, v8::internal::EmbedderStackStateScope::Origin, cppgc::EmbedderStackState) [.nvm/versions/node/v18.16.0/bin/node]
 5: 0x1011e8b6c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [.nvm/versions/node/v18.16.0/bin/node]
 6: 0x1011dce14 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [.nvm/versions/node/v18.16.0/bin/node]
 7: 0x1011dd644 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [.nvm/versions/node/v18.16.0/bin/node]
 8: 0x1011c3478 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [.nvm/versions/node/v18.16.0/bin/node]
 9: 0x1015532bc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [.nvm/versions/node/v18.16.0/bin/node]
10: 0x10189d04c Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [.nvm/versions/node/v18.16.0/bin/node]
11: 0x101921b6c Builtins_StringToLowerCaseIntl [.nvm/versions/node/v18.16.0/bin/node]
12: 0x106964f5c
13: 0x1018f6c18 Builtins_RegExpReplace [.nvm/versions/node/v18.16.0/bin/node]
14: 0x10188eccc Builtins_StringPrototypeReplace [.nvm/versions/node/v18.16.0/bin/node]
15: 0x106769fc8
16: 0x1082aeef8
17: 0x10661f8cc
18: 0x1082674a8
19: 0x1068887e0
20: 0x1068a3070
21: 0x10661c994
22: 0x1082b962c
23: 0x1082cd740
24: 0x1082dba38
25: 0x10649b0cc
26: 0x1082a454c
27: 0x1082ae120
28: 0x10689c040
29: 0x1068e3af4
30: 0x1083f2fcc
31: 0x1068ad374
32: 0x10662621c
33: 0x1065d03c8
34: 0x10673a02c
35: 0x1065d1970
36: 0x10673a02c
37: 0x1065b3bd4
38: 0x1068d9160
39: 0x1065b4770
40: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
41: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
42: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
43: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
44: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
45: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
46: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
47: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
48: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
49: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
50: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
51: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
52: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
53: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
54: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
55: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
56: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
57: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
58: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
59: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
60: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
61: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
62: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
63: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
64: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
65: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
66: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
67: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
68: 0x101828198 Builtins_InterpreterEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
69: 0x1018264d0 Builtins_JSEntryTrampoline [.nvm/versions/node/v18.16.0/bin/node]
70: 0x101826164 Builtins_JSEntry [.nvm/versions/node/v18.16.0/bin/node]
71: 0x10116b85c v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [.nvm/versions/node/v18.16.0/bin/node]
72: 0x10116ad90 v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, int, v8::internal::Handle<v8::internal::Object>*) [.nvm/versions/node/v18.16.0/bin/node]
73: 0x10105b124 v8::Function::Call(v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) [.nvm/versions/node/v18.16.0/bin/node]
74: 0x100f56290 node::Realm::ExecuteBootstrapper(char const*, std::__1::vector<v8::Local<v8::Value>, std::__1::allocator<v8::Local<v8::Value>>>*) [.nvm/versions/node/v18.16.0/bin/node]
75: 0x100eb32a8 node::StartExecution(node::Environment*, char const*) [.nvm/versions/node/v18.16.0/bin/node]
76: 0x100eb31f0 node::StartExecution(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [.nvm/versions/node/v18.16.0/bin/node]
77: 0x100e3c698 node::LoadEnvironment(node::Environment*, std::__1::function<v8::MaybeLocal<v8::Value> (node::StartExecutionCallbackInfo const&)>) [.nvm/versions/node/v18.16.0/bin/node]
78: 0x100f268bc node::NodeMainInstance::Run() [.nvm/versions/node/v18.16.0/bin/node]
79: 0x100eb6028 node::LoadSnapshotDataAndRun(node::SnapshotData const**, node::InitializationResult const*) [.nvm/versions/node/v18.16.0/bin/node]
80: 0x100eb62b4 node::Start(int, char**) [.nvm/versions/node/v18.16.0/bin/node]
81: 0x1a355ff28 start [/usr/lib/dyld]
zsh: abort      NODE_OPTIONS=--max-old-space-size=4096 npx eslint --debug .
justin-taylor-foodhub commented 1 year ago

If it helps, here's a list of rules run on my repo with relative timing perf:

"eslint": "^8.45.0",
"eslint-plugin-deprecation": "^1.4.1"
Rule Time (ms) Relative
deprecation/deprecation 383.058 83.7%
@typescript-eslint/no-unused-vars 30.740 6.7%
no-unexpected-multiline 3.609 0.8%
no-misleading-character-class 3.235 0.7%
no-regex-spaces 2.647 0.6%
no-global-assign 2.325 0.5%
@typescript-eslint/no-loss-of-precision 2.304 0.5%
@typescript-eslint/ban-types 2.134 0.5%
no-control-regex 1.858 0.4%
@typescript-eslint/ban-ts-comment 1.653 0.4%
gund commented 1 year ago

Thanks for posting more details on this!

Regarding the execution time of this rule we already have an issue opened but I'm afraid there is not much we can do judging by this comment a TS API that's used is simply expensive so it's unavoidable and will take time to run this rule on the whole codebase.

But as for memory side of things there is a chance that the rule is doing something that retains a bunch of stuff in memory so we might figure it out I hope.

Anyone is welcome to take a look at the rule for potential memory retention issues btw.

gund commented 1 year ago

Just had a quick glance at the rule code and it seems that most of the code is stateless (with an exception to a small string variable) which means that there are no easy fixes that we can make and most likely the memory consumptions comes directly from the Typescript tooling that the rule is using. So it's pretty much the same story as with the performance issue mentioned above.

One thing we may take a look into is the calls that the rules makes to ESlint APIs like these:

https://github.com/gund/eslint-plugin-deprecation/blob/df4dc2319d058ac0b367d71abe7585624b9510ce/src/rules/deprecation.ts#L85 and https://github.com/gund/eslint-plugin-deprecation/blob/df4dc2319d058ac0b367d71abe7585624b9510ce/src/rules/deprecation.ts#L201

And maybe if it's okay to cache such calls we can save a bit on the performance and possibly even on memory but it largely depends on how those APIs are implemented and what they are doing.

If anyone if more familiar with how ESLintUtils.getParserServices(context) and/or TS program.getTypeChecker() works and maybe can shed more light how expensive is it to call it and if it's possible to cache it at all as it depends on the context.

Also maybe it's a good idea to open similar issue at the upstream repo of @typescript-eslint/utils as it is providing these APIs and defines how they are created/accessed and most likely that's where the issue lies.

karlhorky commented 1 year ago

cc @bradzacher or @jakebailey in case you would have any insights into these memory / performance things here and in #44

jakebailey commented 1 year ago

On the OOM front, if you run node with --heapsnapshot-near-heap-limit you may get back some sort of snapshot that provides insights into what's holding onto all of the memory.

On the general time-based performance front, you can try running pprof-it (https://github.com/jakebailey/pprof-it) and then throw its output into https://www.speedscope.app/ to get a glimpse as to where the time is being spent.

bradzacher commented 1 year ago

If I had to guess - https://github.com/typescript-eslint/typescript-eslint/issues/1192

This plugin is going to be a big candidate to bloat memory because all information is calculated lazily - so computing types for properties, calls, imports, etc so you can grab the JSDoc will load more types than most setups will use normally.