fetch failed https://github.com/isaacs/readable-stream/archive/master.tar.gz

[GDreyV]

Hello, today while building our applications we faced an error:

npm ERR! fetch failed https://github.com/isaacs/readable-stream/archive/master.tar.gz
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 404

We can't change this dependency as it's deeply nested dependency of gulp-cli. Are there any workarounds for it? Node version is v6.17.1 and npm is v3.10.10

[mcollina]

what version of this library are you using? I think that's deeply deprecated.

[GDreyV]

Npm says something like that:

`-- gulp-cli@2.3.0 
  ...
  | +-- readable-stream@2.3.7 

[GDreyV]

I managed to install gulp-cli on node 14.17 but got the same error installing other gulp libraries. Unfortunately we can't move from node 6 as it's inapp plugin which uses node 6 internally.

[mcollina]

I don't know how to help really. That is pretty old code that I'm not sure how it even worked there.

[GDreyV]

Is it possible to restore repository in old location? Why it was removed? It looks like it was there about a month ago.

[GDreyV]

It looks like url I shared redirects to https://codeload.github.com/nodejs/readable-stream/tar.gz/master which returns 404.

[mcollina]

I think we broke you when we renamed the branch from master to main.

[mhdawson]

This may be a case where github is not redirecting everything from master to main as we expected it to.

[mhdawson]

I searched in https://github.com/gulpjs/gulp-cli and I don't get any hits against readable-stream

[mhdawson]

I installed the reported version of Node.js and installed gulp-cli and then did a search to see if I could find anything that looked like it could reference https://github.com/isaacs/readable-stream/archive/master.tar.gz.

What came up was

hosted-git-info/git-host-info.js:    'tarballtemplate': 'https://{domain}/{user}/{project}/repository/archive.tar.gz?ref={committish}',
hosted-git-info/git-host-info.js:    'pathmatch': /^[/]([^/]+)[/]((?!.*(\/-\/|\/repository\/archive\.tar\.gz\?=.*|\/repository\/[^/]+\/archive.tar.gz$)).*?)(?:[.]git|[/])?$/
hosted-git-info/package.json:  "readme": "# hosted-git-info\n\nThis will let you identify and transform various git hosts URLs between\nprotocols.  It also can tell you what the URL is for the raw path for\nparticular file for direct access without git.\n\n## Example\n\n```javascript\nvar hostedGitInfo = require(\"hosted-git-info\")\nvar info = hostedGitInfo.fromUrl(\"git@github.com:npm/hosted-git-info.git\", opts)\n/* info looks like:\n{\n  type: \"github\",\n  domain: \"github.com\",\n  user: \"npm\",\n  project: \"hosted-git-info\"\n}\n*/\n```\n\nIf the URL can't be matched with a git host, `null` will be returned.  We\ncan match git, ssh and https urls.  Additionally, we can match ssh connect\nstrings (`git@github.com:npm/hosted-git-info`) and shortcuts (eg,\n`github:npm/hosted-git-info`).  Github specifically, is detected in the case\nof a third, unprefixed, form: `npm/hosted-git-info`.\n\nIf it does match, the returned object has properties of:\n\n* info.type -- The short name of the service\n* info.domain -- The domain for git protocol use\n* info.user -- The name of the user/org on the git host\n* info.project -- The name of the project on the git host\n\n## Version Contract\n\nThe major version will be bumped any time…\n\n* The constructor stops accepting URLs that it previously accepted.\n* A method is removed.\n* A method can no longer accept the number and type of arguments it previously accepted.\n* A method can return a different type than it currently returns.\n\nImplications:\n\n* I do not consider the specific format of the urls returned from, say\n  `.https()` to be a part of the contract.  The contract is that it will\n  return a string that can be used to fetch the repo via HTTPS.  But what\n  that string looks like, specifically, can change.\n* Dropping support for a hosted git provider would constitute a breaking\n  change.\n\n## Usage\n\n### var info = hostedGitInfo.fromUrl(gitSpecifier[, options])\n\n* *gitSpecifer* is a URL of a git repository or a SCP-style specifier of one.\n* *options* is an optional object. It can have the following properties:\n  * *noCommittish* — If true then committishes won't be included in generated URLs.\n  * *noGitPlus* — If true then `git+` won't be prefixed on URLs.\n\n## Methods\n\nAll of the methods take the same options as the `fromUrl` factory.  Options\nprovided to a method override those provided to the constructor.\n\n* info.file(path, opts)\n\nGiven the path of a file relative to the repository, returns a URL for\ndirectly fetching it from the githost.  If no committish was set then\n`master` will be used as the default.\n\nFor example `hostedGitInfo.fromUrl(\"git@github.com:npm/hosted-git-info.git#v1.0.0\").file(\"package.json\")`\nwould return `https://raw.githubusercontent.com/npm/hosted-git-info/v1.0.0/package.json`\n\n* info.shortcut(opts)\n\neg, `github:npm/hosted-git-info`\n\n* info.browse(path, fragment, opts)\n\neg, `https://github.com/npm/hosted-git-info/tree/v1.2.0`,\n`https://github.com/npm/hosted-git-info/tree/v1.2.0/package.json`,\n`https://github.com/npm/hosted-git-info/tree/v1.2.0/REAMDE.md#supported-hosts`\n\n* info.bugs(opts)\n\neg, `https://github.com/npm/hosted-git-info/issues`\n\n* info.docs(opts)\n\neg, `https://github.com/npm/hosted-git-info/tree/v1.2.0#readme`\n\n* info.https(opts)\n\neg, `git+https://github.com/npm/hosted-git-info.git`\n\n* info.sshurl(opts)\n\neg, `git+ssh://git@github.com/npm/hosted-git-info.git`\n\n* info.ssh(opts)\n\neg, `git@github.com:npm/hosted-git-info.git`\n\n* info.path(opts)\n\neg, `npm/hosted-git-info`\n\n* info.tarball(opts)\n\neg, `https://github.com/npm/hosted-git-info/archive/v1.2.0.tar.gz`\n\n* info.getDefaultRepresentation()\n\nReturns the default output type. The default output type is based on the\nstring you passed in to be parsed\n\n* info.toString(opts)\n\nUses the getDefaultRepresentation to call one of the other methods to get a URL for\nthis resource. As such `hostedGitInfo.fromUrl(url).toString()` will give\nyou a normalized version of the URL that still uses the same protocol.\n\nShortcuts will still be returned as shortcuts, but the special case github\nform of `org/project` will be normalized to `github:org/project`.\n\nSSH connect strings will be normalized into `git+ssh` URLs.\n\n## Supported hosts\n\nCurrently this supports Github, Bitbucket and Gitlab. Pull requests for\nadditional hosts welcome.\n",
hosted-git-info/README.md:eg, `https://github.com/npm/hosted-git-info/archive/v1.2.0.tar.gz`

Looking in the readme I see:

* info.tarball(opts)

eg, `https://github.com/npm/hosted-git-info/archive/v1.2.0.tar.gz`

which looks like it would be a match.

Looking for master this pops:

git-host.js:    vars.committish = vars.committish || 'master'

What I don't see is any references to a call to 'tarball' in the modules which were installed. Think makes me think that it might be something other than gulp-cli that is triggering the install as this working fine for me:

[midawson@midawson testgulp-cli]$ npm --version
3.10.10
[midawson@midawson testgulp-cli]$ node --version
v6.17.1
[midawson@midawson testgulp-cli]$ npm install gulp-cli
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
/home/midawson/testgulp-cli
└── gulp-cli@2.3.0 

npm WARN enoent ENOENT: no such file or directory, open '/home/midawson/testgulp-cli/package.json'
npm WARN testgulp-cli No description
npm WARN testgulp-cli No repository field.
npm WARN testgulp-cli No README data
npm WARN testgulp-cli No license field.

[mhdawson]

One other suggestion is that a work around might be to update

git-host.js:    vars.committish = vars.committish || 'master'

to change master -> main

[GDreyV]

@mhdawson is there any way to see why npm installing the package during the npm i? I mean when they are not installed yet? -ddd doesn't really show dependencies tree on error. Sorry, I didn't get, where should I change git-host.js?

[mhdawson]

I'm not sure if you can see this but I've opened this ticket to report the failure to redirect from master to main to GitHub - https://support.github.com/ticket/personal/0/1348447

[mhdawson]

@GDreyV when I run npm install gulp-cli. The file I saw the matches in was node_modules/hosted-git-info/git-host.js.

[GDreyV]

Yep, you are right, it's not gulp-cli. Dependencies chain is: gulp-htmlbuild@0.3.0 lazy-flatten-stream@0.1.0 proxy-stream@0.2.7 read-write-stream@0.1.7 read-stream@0.5.2 - https://github.com/Raynos/read-stream/blob/3c8ddd81c08b0af0764e83b2905ac37a62119d26/package.json readable-stream@2.2.6

[GDreyV]

I'm not sure if you can see this but I've opened this ticket to report the failure to redirect from master to main to GitHub - https://support.github.com/ticket/personal/0/1348447

Yes, I can't see the issue you created. Could you please update current one when it's fixed or there are some updates?

[mhdawson]

Interestingly

[midawson@midawson testgulp-cli]$ npm install read-stream@0.5.2 
npm ERR! git rev-list -n1 master: fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
npm ERR! git rev-list -n1 master: Use '--' to separate paths from revisions, like this:
npm ERR! git rev-list -n1 master: 'git <command> [<revision>...] -- [<file>...]'
npm ERR! git rev-list -n1 master: 
npm ERR! Linux 4.18.0-305.12.1.el8_4.x86_64
npm ERR! argv "/home/midawson/testgulp-cli/node-v6.17.1-linux-x64/bin/node" "/home/midawson/testgulp-cli/node-v6.17.1-linux-x64/bin/npm" "install" "read-stream@0.5.2"
npm ERR! node v6.17.1
npm ERR! npm  v3.10.10
npm ERR! code 128

npm ERR! Command failed: git rev-list -n1 master
npm ERR! fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
npm ERR! Use '--' to separate paths from revisions, like this:
npm ERR! 'git <command> [<revision>...] -- [<file>...]'
npm ERR! 
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /home/midawson/testgulp-cli/npm-debug.log
[midawson@midawson testgulp-cli]$ npm --version

[GDreyV]

With npm 6.14.13 npm install read-stream@0.5.2 -ddd completed without any errors. Maybe renaming from master to main is supported starting from specific npm version?

[mhdawson]

@MylesBorins is that something that was updated in npm?

[mhdawson]

One of the modules in the path has a hard coded path to https://github.com/isaacs/readable-stream/archive/master.tar.gz

https://github.com/Raynos/read-write-stream/blob/cf26d6fef555e37ba90787a6004974c4ab821d25/package.json#L20

[mhdawson]

Anyway hopefully GitHub will fix up the redirects, but the code pointing to the master in git is already a case of a miracle that it still works.

[mcollina]

Maybe @rayanos could ship a new patch of that!

[kanongil]

Can't this be fixed by creating the master branch again and populate it with an appropriate version of the code? And maybe change README.md to report the reason it is there, pointing to this issue?.

Primary issue with this would be that existing readable-stream developers cannot rely on the master => main branch redirect, and will have to update their master tracking branch manually / remember to pull from main.

[kanongil]

Maybe @rayanos could ship a new patch of that!

Just pinging the correct @Raynos .

[Raynos]

I'll publish a new patch version

[Raynos]

I published 0.1.8

[Raynos]

npm install gulp-cli@2.3.0 -D
// lots of stuff
added 265 packages in 5s

This now works for me, i did not try it before published 0.1.8

[Raynos]

My transitive dependency of gulp-cli worked for the last 9 years untouched...

[mcollina]

Thanks! We renamed the branch from master to main and apparently github missed a redirect.

[mhdawson]

I still think we need GitHub to fix up the redirect as this can affect any project trying to do the right thing by updating their primary branch name but it's great @Raynos could publish an update so quickly to help resolve.

[benjamingr]

Just wanted to hop in and thank Jake personally for promptly taking action to help impacted users while the underlying issue is fixed :)

[GDreyV]

Interestingly

[midawson@midawson testgulp-cli]$ npm install read-stream@0.5.2 
npm ERR! git rev-list -n1 master: fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
npm ERR! git rev-list -n1 master: Use '--' to separate paths from revisions, like this:
npm ERR! git rev-list -n1 master: 'git <command> [<revision>...] -- [<file>...]'
npm ERR! git rev-list -n1 master: 
npm ERR! Linux 4.18.0-305.12.1.el8_4.x86_64
npm ERR! argv "/home/midawson/testgulp-cli/node-v6.17.1-linux-x64/bin/node" "/home/midawson/testgulp-cli/node-v6.17.1-linux-x64/bin/npm" "install" "read-stream@0.5.2"
npm ERR! node v6.17.1
npm ERR! npm  v3.10.10
npm ERR! code 128

npm ERR! Command failed: git rev-list -n1 master
npm ERR! fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
npm ERR! Use '--' to separate paths from revisions, like this:
npm ERR! 'git <command> [<revision>...] -- [<file>...]'
npm ERR! 
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /home/midawson/testgulp-cli/npm-debug.log
[midawson@midawson testgulp-cli]$ npm --version

@mhdawson, sorry, missed this message. How did you fix that? Did you manage to install read-stream somehow?

[mhdawson]

@GDreyV I was working through the issues until I hit: https://github.com/nodejs/readable-stream/issues/463#issuecomment-938209298

Before that I had updated this line

var checkout = parsed.hash && parsed.hash.substr(1) || 'main'

in /home/midawson/testgulp-cli/node-v6.17.1-linux-x64/lib/node_modules/npm/node_modules/normalize-git-url/normalize-git-url.js

I think I also had to remove the cached entry in ~/.npm/read-stream/0.5.2

[GDreyV]

Not sure if I can do it on the build server :( Do you know what the reason of the error? read-stream depends only on git://github.com/Raynos/readable-stream#read-stream which has read-stream branch in it. So I don't follow why does it fail? I mean parsed.hash must have value so it won't return main or master.

[mhdawson]

@MylesBorins and @bnb are still trying to get the issue that was opened with GitHub prioritized - https://support.github.com/ticket/personal/0/1348447

[MylesBorins]

We turned on an experimental feature on the Node.js org so https://github.com/isaacs/readable-stream/archive/master.tar.gz now redirects as expected. If we run into any other issues due to this change we may need to roll it back, we are currently figuring out how we can roll this out for all of dot-com

[GDreyV]

I'm not sure if it's related but I'm still facing the error:

fatal: ambiguous argument 'master': unknown revision or path not in the working tree.

[mhdawson]

The redirect works fine for me, @GDreyV I think you'll need to debug further to figure out/gather more info on the current problem you are seeing.