@nrwl/linter: heap error when linting 2800+ files

[Hotell]

Expected Behavior

lint should work consistently no matter if executed via linter builder or directly via eslint cli

Current Behavior

with following workspace configuration

{
"pb-legacy": {
      "root": "src",
      "sourceRoot": "src/js",
      "projectType": "library",
      "schematics": {},
      "architect": {
        "lint-broken": {
          "builder": "@nrwl/linter:lint",
          "options": {
            "linter": "eslint",
            "config": ".eslintrc-pb-legacy",
            "tsConfig": ["tsconfig.eslint.pb-legacy.json"],
            "exclude": ["**/node_modules/**", "!src/js/**"]
          }
        },
        "lint": {
          "builder": "@nrwl/workspace:run-commands",
          "options": {
            "commands": [
              {
                "command": "eslint --ext .js,.ts,.tsx --config .eslintrc-pb-legacy src/js"
              }
            ]
          }
        }
    }
}

1. 🚨 yarn nx run pb-legacy:lint-broken throw error because of out of memory

<--- Last few GCs --->

[41332:0x103b12000]    79269 ms: Scavenge 1362.1 (1423.2) -> 1361.2 (1423.7) MB, 2.5 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 
[41332:0x103b12000]    79276 ms: Scavenge 1362.1 (1423.7) -> 1361.2 (1424.2) MB, 2.7 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 
[41332:0x103b12000]    79283 ms: Scavenge 1362.4 (1424.2) -> 1361.8 (1425.2) MB, 2.8 / 0.0 ms  (average mu = 0.341, current mu = 0.292) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x101e4d4dbe3d]
    1: StubFrame [pc: 0x101e4e52d6cd]
Security context: 0x31dba791e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x31db3fcce2d1] [/Users/hotell/Projects/devel/productboard/pb-frontend/node_modules/eslint/lib/linter/linter.js:1] [bytecode=0x31dbccfcb971 offset=0](this=0x31db56a026f1 <undefined>,ruleId=0x31dbcd459771 <String[17]: constructor-super>)
    3: arguments adaptor frame: 3->1
   ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10096d6b6 node::Abort() (.cold.1) [/usr/local/Cellar/node@10/10.18.1/bin/node]
 2: 0x10003b9e6 node_module_register [/usr/local/Cellar/node@10/10.18.1/bin/node]
 3: 0x10003bba7 node::OnFatalError(char const*, char const*) [/usr/local/Cellar/node@10/10.18.1/bin/node]

few debug logs from builder:

 nx run pb-legacy:lint-broken

builder options:
   { 
    linter: 'eslint',
     config: '.eslintrc-pb-legacy',
     tsConfig: [ 'tsconfig.eslint.pb-legacy.json' ],
     exclude: [ '**/node_modules/**', '!src/js/**' ],
     format: 'stylish',
     files: [],
     force: false,
     silent: false,
     fix: false,
     cache: false,
     cacheLocation: undefined,
     outputFile: undefined 
}

Linting "pb-legacy"...

- eslintConfigPath: '/Users/hotell/Projects/devel/my-project/.eslintrc-pb-legacy',
- tsConfigs: [ 'tsconfig.eslint.pb-legacy.json' ]
- filesCount: 2839

2. ✅ yarn nx run pb-legacy:lint properly lints codebase

Context

@nrwl/angular : Not Found
  @nrwl/cli : 9.1.2
  @nrwl/cypress : 9.1.2
  @nrwl/eslint-plugin-nx : 9.1.2
  @nrwl/express : Not Found
  @nrwl/jest : 9.1.2
  @nrwl/linter : 9.1.2
  @nrwl/nest : Not Found
  @nrwl/next : Not Found
  @nrwl/node : 9.1.2
  @nrwl/react : 9.1.2
  @nrwl/schematics : Not Found
  @nrwl/tao : 9.1.2
  @nrwl/web : 9.1.2
  @nrwl/workspace : 9.1.2
  typescript : 3.8.3

[hakalb]

I have also experienced this error and finally came up with a solution. The problem origin from how TypeScript and ESLint works together, providing the files to lint.

When creating a new workspace with Nx everything works, since there are limited number of files to lint. As the code base grows this will eventually become a problem.

Suggested fix

To support enterprise workspaces I did the following:

Created a new file tsconfig.eslint.json containing a reference to tsconfig.base.json and the paths that should support linting. For a default Nx workspace, paths would be apps and libs.

{
  "extends": "./tsconfig.base.json",
  "include": [
    "./apps/**/*",
    "./libs/**/*"
  ]
}

Then change parserOptions in .eslintrc to point to the new file.

{
  ...
  "parserOptions": {
    "ecmaVersion": 2020,
    "sourceType": "module",
    "project": "./tsconfig.eslint.json"
  }
  ...
}

The lint block in angular.json should also look something like this.

{
  ...
  "lint": {
    "builder": "@nrwl/linter:lint",
    "options": {
      "linter": "eslint",
      "tsConfig": [
        "libs/shared/ui/loading-message/tsconfig.lib.json",
        "libs/shared/ui/loading-message/tsconfig.spec.json"
      ],
      "exclude": [
        "**/node_modules/**",
        "!libs/shared/ui/loading-message/**/*"
      ]
    }
  ...
}

Now everything should work as expected. Hopefully it will help someone else struggling getting it to work.

Context

Nx version 10.0.12.

Angular CLI: 10.0.6
Node: 12.17.0
OS: darwin x64

Angular: 10.0.9
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker
Ivy Workspace: Yes

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.1000.5
@angular-devkit/build-angular      0.1000.6
@angular-devkit/build-optimizer    0.1000.6
@angular-devkit/build-webpack      0.1000.6
@angular-devkit/core               10.0.6
@angular-devkit/schematics         10.0.6
@angular/cdk                       10.1.3
@angular/cli                       10.0.6
@angular/flex-layout               10.0.0-beta.32
@angular/material                  10.1.3
@angular/material-moment-adapter   10.1.3
@ngtools/webpack                   10.0.6
@schematics/angular                10.0.6
@schematics/update                 0.1000.6
rxjs                               6.6.2
typescript                         3.9.7
webpack                            4.43.0

Finally

Kudos to this issue that got me on track by explaining the problem in more detail.

[JamesHenry]

This situation should be greatly helped by the new builder coming in https://github.com/nrwl/nx/pull/3763

[Hotell]

FYI this OOM error is mainly caused by nx error boundaries rule rather than current linter builder implementation.

[kamiljano]

that didn't help much. I upgraded to @nrwl/*: 11.0.2 and I'm still experiencing the same issue, although my project is significantly smaller than the project the OP is describing. It only has ~350 files in total, but the linting always crashes on my e2e module with ~40 TS files in it and like 5 JSON files.

[Hotell]

As I mentioned, what is slowing things down by huge amount is implementation of nx module boundaries eslint plugin. Try to turn that rule off @kamiljano ✌️. There were no changes in the implementation in v11.

Regarding new eslint builder, while the implementation is better than the old builder, there are no real perf improvements ( we migrated 180 projects nx workspace ).

[tmtron]

As a workaround we increased the memory limit: export NODE_OPTIONS=--max_old_space_size=8000

But after enabling additional checks we still ran into OOM issues.
Now we changed our scripts in package.json:

    "lint:nx-workspace": "./node_modules/.bin/nx workspace-lint",
    "lint:nx": "./node_modules/.bin/nx run-many --all --target=lint",
    "lint": "run-s lint:nx-workspace lint:nx",

AFAIK, run-many will start a separate node-process to lint each project (--all)
Now the execution is slower, but it seems to work.

[hakalb]

I have also experienced this error and finally came up with a solution. The problem origin from how TypeScript and ESLint works together, providing the files to lint.

When creating a new workspace with Nx everything works, since there are limited number of files to lint. As the code base grows this will eventually become a problem.

Suggested fix

To support enterprise workspaces I did the following:

Created a new file tsconfig.eslint.json containing a reference to tsconfig.base.json and the paths that should support linting. For a default Nx workspace, paths would be apps and libs.

{
  "extends": "./tsconfig.base.json",
  "include": [
    "./apps/**/*",
    "./libs/**/*"
  ]
}

Then change parserOptions in .eslintrc to point to the new file.

{
  ...
  "parserOptions": {
    "ecmaVersion": 2020,
    "sourceType": "module",
    "project": "./tsconfig.eslint.json"
  }
  ...
}

The lint block in angular.json should also look something like this.

{
  ...
  "lint": {
    "builder": "@nrwl/linter:lint",
    "options": {
      "linter": "eslint",
      "tsConfig": [
        "libs/shared/ui/loading-message/tsconfig.lib.json",
        "libs/shared/ui/loading-message/tsconfig.spec.json"
      ],
      "exclude": [
        "**/node_modules/**",
        "!libs/shared/ui/loading-message/**/*"
      ]
    }
  ...
}

Now everything should work as expected. Hopefully it will help someone else struggling getting it to work.

Context

Nx version 10.0.12.

Angular CLI: 10.0.6
Node: 12.17.0
OS: darwin x64

Angular: 10.0.9
... animations, common, compiler, compiler-cli, core, forms
... platform-browser, platform-browser-dynamic, platform-server
... router, service-worker
Ivy Workspace: Yes

Package                            Version
------------------------------------------------------------
@angular-devkit/architect          0.1000.5
@angular-devkit/build-angular      0.1000.6
@angular-devkit/build-optimizer    0.1000.6
@angular-devkit/build-webpack      0.1000.6
@angular-devkit/core               10.0.6
@angular-devkit/schematics         10.0.6
@angular/cdk                       10.1.3
@angular/cli                       10.0.6
@angular/flex-layout               10.0.0-beta.32
@angular/material                  10.1.3
@angular/material-moment-adapter   10.1.3
@ngtools/webpack                   10.0.6
@schematics/angular                10.0.6
@schematics/update                 0.1000.6
rxjs                               6.6.2
typescript                         3.9.7
webpack                            4.43.0

Finally

Kudos to this issue that got me on track by explaining the problem in more detail.

Since I upgraded to Nx 11 I'm no longer using my own workaround. tsconfig.eslint.json is removed and the setup is as suggested by the Nrwl team.

This might not help you guys still struggling but since I suggested a workaround I just want to let you know my status. Still though, performance hasn't improved.

[JamesHenry]

I am going to close this one for now, the tooling and configuration has changed quite a bit since this was originally opened, and it is no longer a good foundation for clear discussion on any current issues because of that.

If you encounter performance issues with your ESLint setup in the latest version of Nx, please have a read of: https://github.com/typescript-eslint/typescript-eslint/blob/master/docs/getting-started/linting/FAQ.md#my-linting-feels-really-slow

This will give you a better understanding of the moving pieces involved in using TypeScript and ESLint together. If, on top of that, Nx is causing issues/has bugs we can work on those on a case by case basis.

If that last point applies to you, please open a fresh issue with as much information about your setup as possible: versions of packages, configuration files, folder structures etc. Naturally the best possible submissions include links to repos which reproduce the issues.

Thanks all 🙏

[JamesHenry]

Please do not +1 or "me too" performance issue reports - we need to be able investigate specific setups

[daton89]

@JamesHenry the link you posted lead to a 404 page, can you provide an updated link?

[JamesHenry]

@daton89 sorry, we transitioned from github docs to a full website: https://typescript-eslint.io/docs/linting/troubleshooting#my-linting-feels-really-slow

[github-actions[bot]]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.