Support question - does the NPM registry respond with tarballs? when?

[the1mills]

twimc:

I have this question on SO, and I am afraid it might not get an answer: https://stackoverflow.com/questions/50359954/determine-when-npm-is-making-a-request-for-a-tarball-from-the-registry

I set up a local server to act as a local NPM registry, using:

npm set registry http://localhost:7777

however, when I do an install with a clean cache, my registry server only seems to get requests for JSON. It never seems to get a request for a tarball or actual package data.

My question is - are there some headers in the request to my registry server that I can read to determine if the request is for a tarball?

[the1mills]

I have a feeling that registry.npmjs.org only responds with meta information about packages - there must be some other endpoint X that responds with tarballs. Is there some way for me to proxy endpoint X locally?

[sompylasar]

Yes, it responds with tgz when asked, and this is easy to find out by asking registry.npmjs.org for the package.json metadata which, besides the regular fields, contains the URL to the tgz:

npx get-package-json-from-registry react@16

The response JSON contains dist.tarball of "https://registry.npmjs.org/react/-/react-16.3.2.tgz".

See https://github.com/kesla/download-package-tarball, https://github.com/kesla/get-package-json-from-registry, https://github.com/kesla/get-npm-registry-package

Yarn, for example, by default uses registry.yarnpkg.com server which works as a proxy to registry.npmjs.org as mentioned here: https://github.com/yarnpkg/yarn/issues/889#issuecomment-253237639

You might want to also look at https://github.com/verdaccio/verdaccio, https://github.com/krakenjs/kappa, and https://github.com/kesla/offline-npm

[the1mills]

Your second part makes sense, but the first part confuses me.

it looks like GET requests to registry.npmjs.org/<pkg> only respond with JSON. I am looking to intercept requests for tgz files by using my local registry, not looking to intercept JSON requests.

In other words, when I set up my own registry server, I never see any requests to:

registry.npmjs.org/<pkg>/.../<version.tgz>

none of the urls end in .tgz, as far as I see. But I have completely cleared the cache, so NPM must be getting the packages from somewhere.

[sompylasar]

curl -v https://registry.npmjs.org/react/-/react-16.3.2.tgz
*   Trying 151.101.52.162...
* Connected to registry.npmjs.org (151.101.52.162) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: npmjs.org
* Server certificate: GlobalSign Extended Validation CA - SHA256 - G3
* Server certificate: GlobalSign
> GET /react/-/react-16.3.2.tgz HTTP/1.1
> Host: registry.npmjs.org
> User-Agent: curl/7.49.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
< Server: UploadServer
< Cache-Control: max-age=432000
< Last-Modified: Tue, 29 Sep 2009 17:21:33 GMT
< ETag: "4ac2421d-859d"
< x-npm-region: US-West
< Content-Length: 34205
< Accept-Ranges: bytes
< Date: Wed, 16 May 2018 01:13:38 GMT
< Via: 1.1 varnish
< Age: 61987
< Connection: keep-alive
< X-Served-By: cache-sea1049-SEA
< X-Cache: HIT
< X-Cache-Hits: 137
< X-Timer: S1526433218.479861,VS0,VE0
< 

This was a GET request to registry.npmjs.org, and it responded with application/octet-stream (which was the binary tgz in the response body that I omitted here).

So your particular issue is not that the server does not respond with a tgz, but that these tgz requests either aren't executed (because of a local cache), or aren't reaching your local proxy that you configured as a registry.

A. Which exact commands do you run to 1) clear the npm caches; 2) install a package to trigger the registry requests?

B. What are you trying to achieve with this? Why aren't you using the already implemented npm proxies?

[leofisG]

When you do npm install <package-name>. There are actually two GET requests:

1. get the JSON (as you said) which is a manifest that contains all the metadata for a given package, example here.
2. get the actual tarball via the JSON manifest (see dist.tarball)