search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.61k stars 2.36k forks source link

Post-upgrade nx16 => nx18 dep-graph to file super slow #22700

Open work933k opened 7 months ago

work933k commented 7 months ago

Current Behavior

I'm doing a trial NX upgrade from v16.3.2 to v18.2.1 and notice that doing npx nx dep-graph --file=dist/dep-graph/dep-graph.html has become much slower. It now takes over 7 minutes, while it used to take 4 seconds.

Expected Behavior

I would not expect such a significant increase of time.

GitHub Repo

No response

Steps to Reproduce

1.

Nx Report

NX   Report complete - copy this into the issue template

Node   : 18.16.0
OS     : linux-x64
npm    : 9.5.1

nx                 : 18.2.1
@nx/js             : 18.2.1
@nx/jest           : 18.2.1
@nx/linter         : 18.2.1
@nx/eslint         : 18.2.1
@nx/workspace      : 18.2.1
@nx/angular        : 18.2.1
@nx/cypress        : 18.2.1
@nx/devkit         : 18.2.1
@nx/eslint-plugin  : 18.2.1
@nx/express        : 18.2.1
@nx/node           : 18.2.1
@nx/plugin         : 18.2.1
@nx/storybook      : 18.2.1
@nrwl/tao          : 18.2.1
@nx/web            : 18.2.1
@nx/webpack        : 18.2.1
typescript         : 5.4.3
---------------------------------------
Community plugins:
@compodoc/compodoc     : 1.1.23
@ngrx/effects          : 17.0.1
@ngrx/entity           : 17.0.1
@ngrx/eslint-plugin    : 17.0.1
@ngrx/router-store     : 17.0.1
@ngrx/schematics       : 17.0.1
@ngrx/store            : 17.0.1
@ngrx/store-devtools   : 17.0.1
@spaceribs/nx-betterer : 2.0.2
@storybook/angular     : 7.6.17
ng-mocks               : 14.12.1
nx-stylelint           : 17.1.4
---------------------------------------
Local workspace plugins:
         @dummy-nx/storybook-test-runner
         @dummy-nx/static-server
         @dummy-nx/azuredevops
         @dummy-nx/chromatic
         @dummy-nx/workspace-plugin

Failure Logs

No response

Package Manager Version

npm v9.5.1

Operating System

Additional Information

WSL2. I'll try to gather more info and it to this issue.

As you can see below it takes 7minutes to generate the HTML. To generate a json file takes less than 1second. Something wierd is going on...

root@dtl2022q4:~/DummyNx# time npx nx dep-graph --file=dist/dep-graph/dep-graph.html       

 NX   HTML output created in /root/DummyNx/dist/dep-graph

/root/DummyNx/dist/dep-graph
/root/DummyNx/dist/dep-graph/static/3rdpartylicenses.txt
/root/DummyNx/dist/dep-graph/static/environment.js
/root/DummyNx/dist/dep-graph/static/favicon.ico
/root/DummyNx/dist/dep-graph/static/main.js
/root/DummyNx/dist/dep-graph/static/runtime.js
/root/DummyNx/dist/dep-graph/static/styles.css
/root/DummyNx/dist/dep-graph/static/styles.js

real    7m7.574s
user    7m32.900s
sys     0m13.846s
==============================
root@dtl2022q4:~/DummyNx# time npx nx dep-graph --file=dist/dep-graph/dep-graph.json 

 NX   JSON output contains deprecated fields:

- affectedProjects
- criticalPath

These fields will be removed in Nx 19. If you need to see which projects were affected, use `nx show projects --affected`.

 NX   JSON output created in /root/DummyNx/dist/dep-graph

/root/DummyNx/dist/dep-graph/dep-graph.json

real    0m0.870s
user    0m0.919s
sys     0m0.144s

If i run NODE_DEBUG=* npx nx dep-graph --file=dist/dep-graph/dep-graph.html then i see no output for a long time after this:

STREAM 277839: readableAddChunk <Buffer 6c 69 62 73 2f 73 68 61 72 65 64 2f 6d 61 74 65 72 69 61 6c 2d 74 68 65 6d 65 2d 7a 77 69 74 73 65 72 6c 65 76 65 6e 2f 73 72 63 2f 6c 69 62 2f 73 74 ... 27821 more bytes>
STREAM 277839: maybeReadMore read 0
STREAM 277839: read 0
STREAM 277839: need readable true
STREAM 277839: length less than watermark true
STREAM 277839: do read
NET 277839: _read

I just noticed that the generated environment file is 150mb big:

root@dtl2022q4:~/DummyNx/dist/dep-graph/static# ls -lai -h
total 154M
266531 drwxr-xr-x 2 root root 4.0K Apr  5 20:18 .
266022 drwxr-xr-x 3 root root 4.0K Apr  5 19:56 ..
 96431 -rw-r--r-- 1 root root  33K Apr  5 20:18 3rdpartylicenses.txt
265180 -rw-r--r-- 1 root root 152M Apr  5 20:24 environment.js
265738 -rw-r--r-- 1 root root  15K Apr  5 20:18 favicon.ico
266536 -rw-r--r-- 1 root root 1.9M Apr  5 20:18 main.js
267086 -rw-r--r-- 1 root root 1.5K Apr  5 20:18 runtime.js
267089 -rw-r--r-- 1 root root  26K Apr  5 20:18 styles.css
267090 -rw-r--r-- 1 root root  121 Apr  5 20:18 styles.js

In the environment.js file is see a continuous repetition of stuff which comes from the nx.json. This is a part of what is see:

window.projectGraphResponse = {
    "hash":"49aed4aab2036c85308131ddcb0be0985ddae3ebebff659db3b49c1cb12e407c",
    "projects":[
        {
            "name":"mzl-employer-libs-mutations-feature-bulk-edit-participants",
            "type":"lib",
            "data":{
                "root":"libs/mzl-employer/libs/mutations/feature/bulk-edit-participants",
                "targets":{
                    "e2e":{
                        "executor":"@nx/cypress:cypress",
                        "inputs":["e2e","^production","{workspaceRoot}/libs/plugins/cypress-preset/index.ts"],
                        "outputs":["{workspaceRoot}/dist/cypress/{projectRoot}","{workspaceRoot}/coverage/cypress/{projectRoot}","{workspaceRoot}/tmp/coverage/cypress/{projectRoot}"],
                        "cache":true,
                        "options":{
                            "cypressConfig":"libs/mzl-employer/libs/mutations/feature/bulk-edit-participants/cypress.config.ts",
                            "testingType":"e2e",
                            "devServerTarget":"mzl-employer-app:serve"
                        },
                        "configurations":{
                            "ci":{
                                "env":{
                                    "CYPRESS_coverage":true
                                },
                                "devServerTarget":"mzl-employer-app:serve-static"
                            },
                            "dummy":{
                                "watch":true,"exit":true
                            },
                            "coverage":{
                                "devServerTarget":"mzl-employer-app:serve:e2e",
                                "env":{
                                    "CYPRESS_coverage":true
                                    }
                                }
                            }
                        },
                        "lint":{
                            "inputs":["lint","{workspaceRoot}/.eslintrc.json","{workspaceRoot}/tools/eslint-rules/**/*"],
                            "outputs":["{options.outputFile}"],
                            "executor":"@nx/eslint:lint",
                            "cache":true,
                            "options":{
...

And that for all targets and for each of the 100 projects.

Is that as expected?

xiongemi commented 7 months ago

@work933k is it possible to provide a link to your repo?

work933k commented 7 months ago

not without losing my job ;-)

zalos commented 5 months ago

Running into the same with a migration from nx 16 to 17. And same as @work933k , I cannot share the repo. And unfortunately it's something unique as we have other areas with nx 17 and nx 18 and those run fine. Any ideas for how to profile it so we can give you some additional information?

Context I can share:

The generated environment.js in the healthy apps (and same repo pre nx 17) are around 3-6MB. The slow repo is generating 214MB

Checking that file. The 3 variables are

So it's something in that area...