Docker Analyzer runs forever for project particle-emitter

[alexandruz]

The OSS Review Toolkit, version e57fed5. docker build

OSS Project: https://github.com/pixijs/particle-emitter/

Found 1 Yarn definition file(s) at:
        package.json
Found 1 definition file(s) from 1 package manager(s) in total.
...
15:09:54.014 [main] INFO  org.ossreviewtoolkit.analyzer.Analyzer - Starting Yarn analysis.
...
15:09:54.418 [DefaultDispatcher-worker-1] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'yarn install --ignore-scripts --ignore-engines' in '/ort'...
15:10:12.999 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - yarn install v1.22.17
15:10:12.999 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - [1/4] Resolving packages...
15:10:12.999 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - [2/4] Fetching packages...
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - [3/4] Linking dependencies...
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - [4/4] Building fresh packages...
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - success Saved lockfile.
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - Done in 18.39s.
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - warning Ignored scripts due to flag.
15:10:13.000 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -
15:10:13.084 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Parsing project info from '/ort/package.json'.

after a couple hundred MB of logs:

17:12:37.115 [DefaultDispatcher-worker-2] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Starting to parse '/ort/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/@pixi/particle-emitter/package.json'...
17:12:37.115 [DefaultDispatcher-worker-27] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Found a 'package.json' file in '/ort/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/@pixi/particle-emitter'.
17:12:37.116 [DefaultDispatcher-worker-27] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - The package directory '/ort/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/@pixi/particle-emitter' links to '/ort'.
17:12:37.116 [DefaultDispatcher-worker-27] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Resolving the package info for 'NPM:@pixi:particle-emitter:5.0.8' locally from '/ort'.
17:12:37.119 [DefaultDispatcher-worker-27] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Finished parsing '/ort/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/@pixi/particle-emitter/package.json' to 'NPM:@pixi:particle-emitter:5.0.8'.

Something is amiss and I can't figure it what/why. 🙏🏻

tried with:

$ ort -P ort.analyzer.allowDynamicVersions=true --debug analyze -i /ort -o /ort -f JSON
$ ort --debug analyze -m yarn -i /ort -o /ort

[sschuberth]

.../node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/...

This looks a lot like a circular dependency. We had issues like this before, but fixed them. So this could be a regression.

[sschuberth]

@fviernau, could it be that this broke in 3b6ad90ade82085d3bb8bfaa293c6d7c7c1dca93 as that removed the log about "Not adding circular dependency '$identifier' to the tree"?

[fviernau]

fviernau, could it be that this broke in 3b6ad90 as that removed the log about "Not adding circular dependency '$identifier' to the tree"?

This commit does not remove the log output, it left it here: https://github.com/oss-review-toolkit/ort/commit/3b6ad90ade82085d3bb8bfaa293c6d7c7c1dca93#diff-28d89c724dd945fe93dec5700d61d4dcdaab02f492e09613c8fee6ac5c76cbe6R397

There have been quite a bit of commits since back then, any special reason besides the mentioned "removal of log output", to investigate it from this point?

[fviernau]

@alexandruz - this can be reproduced by running the analyzer on [1], right?

Apart from that, which params did you provide for the max memory to java ? -Xmx?

[1] https://github.com/pixijs/particle-emitter/

[sschuberth]

There have been quite a bit of commits since back then, any special reason besides the mentioned "removal of log output", to investigate it from this point?

No, I was just searching for removals of "circular" and landed straight there.

[alexandruz]

I didn't change any of the default settings, but I'll try with some more memory. @fviernau yes, those debug messages are from running analyzer on the respective project

[alexandruz]

ort seems to take advantage of entire free memory on the system, but in this case nothing happens, as it can with 16 CPUs and a maximum of 7952 MiB of memory (ec2 instance), but the same behavior as described.

some of the logs:

10:58:47.358 [DefaultDispatcher-worker-35] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Found a 'package.json' file in '/builds/alexandruz/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/table/node_modules/ajv'.
10:58:47.358 [DefaultDispatcher-worker-35] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Finished parsing '/builds/alexandruz/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/pixi-v6-iife/node_modules/@pixi/particle-emitter/node_modules/table/node_modules/ajv/package.json' to 'NPM::ajv:8.6.2'.

[alexandruz]

I've create a test package, similar to particle-emitter:

├── package.json
└── test
    └── my-ort-test-package
        └── package.json

with the following contents of these files:

$ cat package.json
{
  "name": "my-ort-package",
  "version": "1.0.0",
  "workspaces": {
    "packages": [
      "./",
      "test/my-ort-test-package"
    ]
  },
  "private": true
}

cat test/my-ort-test-package/package.json
{
    "name": "my-ort-test-package",
    "version": "1.0.0",
    "dependencies": {
        "my-ort-package": "*"
    },
    "workspaces": {
        "nohoist": [
            "my-ort-package"
        ]
    },
    "private": true
}

And I get a similar result:

08:49:31.945 [DefaultDispatcher-worker-7] DEBUG org.ossreviewtoolkit.plugins.packagemanagers.node.Npm - The package directory '/ort/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package/node_modules/my-ort-test-package/node_modules/my-ort-package' links to '/ort'.

[alexandruz]

hello @fviernau,

testing the created package I realized that there is a circular dependency that ort analyzer doesn't pick up on, because it doesn't manage correctly the workspaces setting with the nohoist options.

Can you provide some help where should I look it the code for this issue?

Thanks.

[sschuberth]

there is a circular dependency that ort analyzer doesn't pick up on, because it doesn't manage correctly the workspaces setting with the nohoist options.

Actually, ORT itself isn't interpreting the nohoist option at all. Instead, we simply call yarn install and parse the resulting node_modules directory hierarchy. That's where the issue is located. I'll propose a fix soon.