PNPM problem in private package resolution

kielingraphael commented 1 year ago

Expectation:

That ort handles private packages using PNPM and .npmrc.

Current:

During the ort analysis the PNPM is not being able to get one of our private packages. ort is throwing 404 taking the npm registry instead of our private one.

I tried

Check if .npmrc was wrong right before the ort command but it was totally fine.
Tried to execute the pnpm/npm view my-private-package@version, right before the ort command and it works. But ort itself was not able to run it properly.

How to debug it? I tried a lot of different things and configurations but I'm not able to understand why the jfrog URL is being changed to npm registry url inside the ort child process.

Our .npmrc that works:

@priv-package:registry=https://x.jfrog.io/xx/api/npm/public-npm/
//x.jfrog.io/xx/api/npm/public-npm/:_authToken=${NPM_TOKEN}

The result inside the logs:

...more logs
\ GET https://registry.npmjs.org/x- Not found\nnpm ERR!\
...more logs

sschuberth commented 1 year ago

Does running pnpm install ... like

https://github.com/oss-review-toolkit/ort/blob/0874da5f36a02c5aa4cc5231068e77dde6ef0ba5/analyzer/src/main/kotlin/managers/Pnpm.kt#L77-L85

manually also work for you?

kielingraphael commented 1 year ago

Yes:

pnpm install ... with all the options is working
pnpm view @my-lib works and shows the correct jfrog url

kielingraphael commented 1 year ago

Also, going deeper without know all the ORT context, checking the stashXXXX folder seems that my library is there.

kielingraphael commented 1 year ago

In the end, i'm getting:

11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - {
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -   "error": {
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -     "code": "E404",
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -     "summary": "'@priv-package/xx' is not in the npm registry.\nYou should bug the author to publish it\n(or use the name yourself!)\n\nNote that you can also install from a\ntarball, folder, http url, or git url.",
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -     "detail": "\n '@priv-package/xx@10.0.0' is not in the npm registry.\nYou should bug the author to publish it (or use the name yourself!)\n\nNote that you can also install from a\ntarball, folder, http url, or git url."
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture -   }
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - }
11:39:33.342 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! code E404
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 '@priv-package/xx' is not in the npm registry.
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 You should bug the author to publish it
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 (or use the name yourself!)
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 Note that you can also install from a
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 tarball, folder, http url, or git url.
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404  '@priv-package/xx@10.0.0' is not in the npm registry.
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 Note that you can also install from a
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! 404 tarball, folder, http url, or git url.
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR! A complete log of this run can be found in:
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - npm ERR!     /.../.npm/_logs/2023-02-22T14_39_33_276Z-debug.log
11:39:33.343 [DefaultDispatcher-worker-21] DEBUG org.ossreviewtoolkit.utils.common.ProcessCapture - 
11:39:33.360 [DefaultDispatcher-worker-1] DEBUG org.ossreviewtoolkit.analyzer.managers.Npm - Unable to get package details from a remote registry: IOException: Running 'pnpm view --json @priv-package/xx@10.0.0' in '.../node_modules/@priv-package/xx' failed with exit code 1:

sschuberth commented 1 year ago

Ah, ok, I believe I get what's going on: We use pnpm just to install the packages. All further processing is done by Npm.kt, which also calls npm view to get additional metadata about the packages from the public NPM registry (not sure if npm view is even supposed to work with private registries). And the latter fails as that private package is not available in the public NPM registry. This somewhat reminds me of https://github.com/oss-review-toolkit/ort/issues/5632.

sschuberth commented 1 year ago

On the other hand, I just read again the commit message of a43d3d1b0bd7d0e4c36937f53d4bb20ac6beff16, which mentions that one of the motivations to switch to npm view was that it does work with private NPM registries, hmm...

kielingraphael commented 1 year ago

You have a very good point, but manually at the root of the project both of them are working npm view and pnpm view

kielingraphael commented 1 year ago

Another thing is that checking the logs it is running using pnpm view --json xx, so seems right. Probably because the Pnpm.kt overrides the command to pnpm.

sschuberth commented 1 year ago

Maybe @MarcelBochtler has some input about this.

MarcelBochtler commented 1 year ago

That ort handles private packages using PNPM and .npmrc.

The PNPM implementation was tested with a private repository, so my expectation is also that this should work.

I just did set up a minimal example and it works for me: package.json

{
  "name": "foobar",
  "version": "1.0.0",
  "dependencies": {
    "@myownnamespace/portal-ui": "^1.0.0"
  }
}

The @myownnamespace dependency is only available in a private repository, which I configured ~/.npmrc:

@myownnamespace:registry=https://private.repository.com/_packaging/ui/npm/registry/
//private.repository.com/_packaging/ui/npm/registry/:username=<user>
//private.repository.com/_packaging/ui/npm/registry/:_password=<password>
//private.repository.com/_packaging/ui/npm/registry/:email=<mail>
//private.repository.com/_packaging/ui/npm/:username=<user>
//private.repository.com/_packaging/ui/npm/:_password=<password>
//private.repository.com/_packaging/ui/npm/:email=<mail>

Running pnpm view @myownnamespace/portal-ui and ort --debug analyze -i . -o ort/ both yield correct results.

Where is the .npmrc located? Your home directory? The project directory? Maybe both?

kielingraphael commented 1 year ago

It's located at the root of the project. Today we have a github action that:

Start the docker
Copy the project to a folder
Create a .npmrc file
Run the analyze
...

If right before the analyze command in the SAME docker process i do:

Run pnpm install, it works
Run pnpm view ... it works
Cat the .npmrc to check the token, it is right

One important point that i forgot to emphasize is that only ONE library is facing this problem (as i can see), we have multiple private libraries and checking the logs all of them seems right.

Example of my .npmrc:

++ @lib-that-works:registry=https://x/npm/public-npm/
-- @lib-that-fails:registry=https://x/npm/public-npm/
++ @lib-that-works2:registry=https://x/npm/public-npm/
++ @lib-that-works3:registry=https://x/npm/public-npm/
   //x/npm/public-npm/:_authToken=${NPM_TOKEN}

sschuberth commented 2 months ago

@MarcelBochtler would you be able to spend some time to check again with the above instructions to reproduce the issue?

oss-review-toolkit / ort

PNPM problem in private package resolution #6535