search

apache / cordova-fetch

Apache Cordova Fetch Library
https://cordova.apache.org/
Apache License 2.0
27 stars 27 forks source link

cordova@8.0.0 -> cordova@9.0.0 breaks plugin installs using git repos or local directories #87

Closed shankari closed 4 years ago

shankari commented 4 years ago

Bug Report

Problem

What is expected to happen?

$ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git

should succeed since "url to a git repository containing a plugin.xml" is a valid plugin-spec.

What does actually happen?

Information

on cordova@8.0.0, plugins can be added using either the registry or a git URL

$ npx cordova --version
8.0.0
$ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git
Installing "phonegap-plugin-contentsync" for android
Installing "phonegap-plugin-contentsync" for ios
Adding phonegap-plugin-contentsync to package.json
Saved plugin info for "phonegap-plugin-contentsync" to config.xml
$ npx cordova plugin rm phonegap-plugin-contentsync
Uninstalling phonegap-plugin-contentsync from android
Uninstalling phonegap-plugin-contentsync from ios
Removing "phonegap-plugin-contentsync"
Removing plugin phonegap-plugin-contentsync from config.xml file...
Removing phonegap-plugin-contentsync from package.json
$ npx cordova plugin add phonegap-plugin-contentsync
Installing "phonegap-plugin-contentsync" for android
Installing "phonegap-plugin-contentsync" for ios
Adding phonegap-plugin-contentsync to package.json
Saved plugin info for "phonegap-plugin-contentsync" to config.xml
$ npx cordova plugin rm phonegap-plugin-contentsync
Uninstalling phonegap-plugin-contentsync from android
Uninstalling phonegap-plugin-contentsync from ios
Removing "phonegap-plugin-contentsync"
Removing plugin phonegap-plugin-contentsync from config.xml file...
Removing phonegap-plugin-contentsync from package.json

In the same directory, with cordova@9.0.0, plugins can only be added using the registry, and only intermittently

$ npm install cordova@9.0.0

+ cordova@9.0.0
added 161 packages from 112 contributors, removed 73 packages, updated 53 packages and audited 1707 packages in 19.859s

Note that, in the case of the registry add, the first run of the command fails, and the second run of the identical command succeeds

$ npx cordova plugin add phonegap-plugin-contentsync
Failed to fetch plugin phonegap-plugin-contentsync via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync

$ npx cordova plugin add phonegap-plugin-contentsync
Installing "phonegap-plugin-contentsync" for android
Installing "phonegap-plugin-contentsync" for ios
Adding phonegap-plugin-contentsync to package.json

However, the git add always fails.

$ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git
Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync

$ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git
Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync

$ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git
Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry.
Probably this is either a connection problem, or plugin spec is incorrect.
Check your connection and plugin name/version/URL.
Could not determine package name from output:
phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync

Command or Code

$ cordova plugin add <git URL>

OR

$ cordova plugin add <local URL>

Environment, Platform, Device

Mac OSX Catalina 10.15.5

Version information

For cordova and ionic:

$ npx cordova --version
9.0.0 (cordova-lib@9.0.1)
$ npx ionic --version
6.10.1

For other cordova dependencies, I have attached my package.json package.json.gz

For the rest of the environment:

export NVM_VERSION=0.35.3
export NODE_VERSION=14.5.0
export NPM_VERSION=6.14.5

Checklist

shankari commented 4 years ago

I poked around this a little bit, and the Failed to fetch plugin ... via registry error is from cordova-lib/src/plugman/fetch.js

Code ``` .catch(function (error) { var message = 'Failed to fetch plugin ' + plugin_src + ' via registry.' + '\nProbably this is either a connection problem, or plugin spec is incorrect.' + '\nCheck your connection and plugin name/version/URL.' + '\n' + error; return Promise.reject(new CordovaError(message)); ```

So the real error is Could not determine package name from output

shankari commented 4 years ago

Could not determine package name from output is from node_modules//cordova-fetch/index.js

Code ``` function getTargetPackageSpecFromNpmInstallOutput (npmInstallOutput) { const lines = npmInstallOutput.split('\n'); for (let i = 0; i < lines.length; i++) { if (lines[i].startsWith('+ ')) { // npm >= 5 return lines[i].slice(2); } else if (lines[i].startsWith('└─') || lines[i].startsWith('`-')) { // 3 <= npm <= 4 return lines[i].slice(4).split(' ')[0]; } } throw new CordovaError('Could not determine package name from output:\n' + npmInstallOutput); } ```

Essentially, the code expects the npm install output to have a +, but it doesn't. The output is just phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync with no +, which is why this fails.

shankari commented 4 years ago

The npm install command is launched using superspawn.spawn.

And now I come to the part that I don't understand. If I use -d, I can see the arguments passed in to npm install

Command line arguments for npm install ``` No scripts found for hook "before_plugin_add". Calling plugman.fetch on plugin "https://github.com/phonegap/phonegap-plugin-contentsync.git" fetch: Installing https://github.com/phonegap/phonegap-plugin-contentsync.git to /Users/kshankar/e-mission/upgrade_platform Running command: npm install https://github.com/phonegap/phonegap-plugin-contentsync.git --production --save Command finished with error code 0: npm install,https://github.com/phonegap/phonegap-plugin-contentsync.git,--production,--save Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry. Probably this is either a connection problem, or plugin spec is incorrect. Check your connection and plugin name/version/URL. Could not determine package name from output: phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync CordovaError: Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry. Probably this is either a connection problem, or plugin spec is incorrect. Check your connection and plugin name/version/URL. Could not determine package name from output: phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync at /Users/kshankar/e-mission/upgrade_platform/node_modules/cordova-lib/src/plugman/fetch.js:146:43 at processTicksAndRejections (internal/process/task_queues.js:97:5) ```

If I run the exact same command using superspawn using the node REPL, I get an output with the +

running same command in node REPL ``` $ node Welcome to Node.js v14.5.0. Type ".help" for more information. > var superspawn = require('cordova-common').superspawn; undefined > var dest = "" undefined > var args = ["install", "https://github.com/phonegap/phonegap-plugin-contentsync.git", "--production", "--save"] undefined > superspawn.spawn('npm', args, { cwd: dest }).then(function(output){ console.log(output); }); Promise { promiseDispatch: [Function (anonymous)], valueOf: [Function (anonymous)], inspect: [Function (anonymous)] } > (node:59879) [DEP0097] DeprecationWarning: Using a domain property in MakeCallback is deprecated. Use the async_context variant of MakeCallback or the AsyncResource class instead. (Use `node --trace-deprecation ...` to show where the warning was created) + phonegap-plugin-contentsync@1.4.2 updated 1 package and audited 1707 packages in 11.991s 47 packages are looking for funding run `npm fund` for details found 173 vulnerabilities (45 low, 75 moderate, 53 high) run `npm audit fix` to fix them, or `npm audit` for details ```

But if I add the same kind of output logging in fetch.js, it returns the output without the +

e.g. if I change the code to

New code ``` function installPackage (target, dest, opts) { return isNpmInstalled() // Ensure that `npm` installs to `dest` and not any of its ancestors .then(_ => fs.ensureDir(path.join(dest, 'node_modules'))) // Run `npm` to install requested package .then(_ => npmArgs(target, opts)) .then(args => { events.emit('verbose', `fetch: Installing ${target} to ${dest}`); return superspawn.spawn('npm', args, { cwd: dest }); }) .then(function(output){ console.log("Output = "+output); return output; }) // Resolve path to installed package .then(getTargetPackageSpecFromNpmInstallOutput) .then(spec => pathToInstalledPackage(spec, dest)); } ```

I get the following output

non-REPL usage

I can't figure out why superspawn behaves differently in the REPL and in the script. Any thoughts?

shankari commented 4 years ago

Finally, -d also gives us a clue as to why the registry install works (eventually). It uses plugman install instead of npm install

Logs ``` $ npx cordova plugin -d add phonegap-plugin-contentsync No scripts found for hook "before_plugin_add". No version specified for phonegap-plugin-contentsync, retrieving version from package.json Calling plugman.fetch on plugin "phonegap-plugin-contentsync@^1.4.2" Copying plugin "/Users/kshankar/e-mission/upgrade_platform/node_modules/phonegap-plugin-contentsync" => "/Users/kshankar/e-mission/upgrade_platform/plugins/phonegap-plugin-contentsync" Calling plugman.install on plugin "/Users/kshankar/e-mission/upgrade_platform/plugins/phonegap-plugin-contentsync" for platform "android Installing "phonegap-plugin-contentsync" for android Running command: /Users/kshankar/e-mission/upgrade_platform/platforms/android/cordova/version Command finished with error code 0: /Users/kshankar/e-mission/upgrade_platform/platforms/android/cordova/version Finding scripts for "before_plugin_install" hook from plugin phonegap-plugin-contentsync on android platform only. No scripts found for hook "before_plugin_install". Install start for "phonegap-plugin-contentsync" on android. PlatformApi successfully found for platform android Beginning processing of action stack for android project... Action stack processing complete. Install complete for phonegap-plugin-contentsync on android. Finding scripts for "after_plugin_install" hook from plugin phonegap-plugin-contentsync on android platform only. No scripts found for hook "after_plugin_install". Calling plugman.install on plugin "/Users/kshankar/e-mission/upgrade_platform/plugins/phonegap-plugin-contentsync" for platform "ios Installing "phonegap-plugin-contentsync" for ios Running command: /Users/kshankar/e-mission/upgrade_platform/platforms/ios/cordova/version Command finished with error code 0: /Users/kshankar/e-mission/upgrade_platform/platforms/ios/cordova/version Finding scripts for "before_plugin_install" hook from plugin phonegap-plugin-contentsync on ios platform only. No scripts found for hook "before_plugin_install". Install start for "phonegap-plugin-contentsync" on ios. PlatformApi successfully found for platform ios Beginning processing of action stack for ios project... Adding non-custom framework to project... libz.dylib -> {"customFramework":false,"embed":false,"link":true,"weak":false} Non-custom framework added to project. libz.dylib -> {"customFramework":false,"link":true,"weak":false} Action stack processing complete. pods.json found in platforms/ios Podfile found in platforms/ios Install complete for phonegap-plugin-contentsync on ios. Finding scripts for "after_plugin_install" hook from plugin phonegap-plugin-contentsync on ios platform only. No scripts found for hook "after_plugin_install". Adding phonegap-plugin-contentsync to package.json No scripts found for hook "after_plugin_add". ```
raphinesse commented 4 years ago

Thanks for reporting this and for the detailed information and debugging. It's indeed strange that the npm output differs between manual invocations and those performed by cordova-fetch.

Just to be sure that this is still an issue in the current code base, could you please try if you can reproduce this issue when using cordova@nightly?

shankari commented 4 years ago

Ok, so I poked around this some more, and I know the reason behind the difference in superspawn behavior.

After adding log statements to all instances of superspawn, it looks like this is difference is due to the differences in the versions of superspawn included in the project.

$ find node_modules -name \*superspawn.js
node_modules/cordova/node_modules/cordova-common/src/superspawn.js
node_modules/cordova-common/src/superspawn.js
node_modules/cordova-create/node_modules/cordova-common/src/superspawn.js
node_modules/cordova-fetch/node_modules/cordova-common/src/superspawn.js
node_modules/cordova-lib/node_modules/cordova-common/src/superspawn.js
node_modules/phonegap/node_modules/cordova-common/src/superspawn.js
node_modules/phonegap/node_modules/cordova-create/node_modules/cordova-common/src/superspawn.js

While using node directly, we use node_modules/cordova-common/src/superspawn.js, while using plugin add, we use node_modules/cordova-fetch/node_modules/cordova-common/src/superspawn.js. The two versions are different, although I cannot see which of the differences lead to this difference in behavior.

not_working_to_working.diff.gz

shankari commented 4 years ago

so the two versions are different because the versions of cordova-common are different.

$ find node_modules -name package.json | grep "cordova-common.package" | xargs grep version.:
node_modules/cordova/node_modules/cordova-common/package.json:  "version": "3.2.1"
node_modules/cordova-common/package.json:  "version": "4.0.2"
node_modules/cordova-create/node_modules/cordova-common/package.json:  "version": "3.2.1"
node_modules/cordova-fetch/node_modules/cordova-common/package.json:  "version": "3.2.1"
node_modules/cordova-lib/node_modules/cordova-common/package.json:  "version": "3.2.1"
node_modules/phonegap/node_modules/cordova-common/package.json:  "version": "2.1.1"
node_modules/phonegap/node_modules/cordova-create/node_modules/cordova-common/package.json:  "version": "2.2.5"
shankari commented 4 years ago

But if I use the cordova-fetch version superspawn directly in the REPL, it works correctly. Which is consistent with the diffs seeming fairly innocuous. The main difference seems to be running this in cordova.

Logs ``` > var superspawn = require('./node_modules/cordova-fetch/node_modules/cordova-common/src/superspawn').superspawn; > superspawn.spawn('npm', args, { cwd: dest }).then(function(output){ console.log(output); return output;}) Promise { promiseDispatch: [Function (anonymous)], valueOf: [Function (anonymous)], inspect: [Function (anonymous)] } > + phonegap-plugin-contentsync@1.4.2 updated 1 package and audited 1707 packages in 19.092s 47 packages are looking for funding run `npm fund` for details found 173 vulnerabilities (45 low, 75 moderate, 53 high) run `npm audit fix` to fix them, or `npm audit` for details ```

I will try with the latest cordova@nightly and if that still doesn't work, I'm going to try to create a MCVE to make it easier for others to debug as well.

shankari commented 4 years ago

@raphinesse Confirmed that this occurs even with cordova@nightly

Logs ``` $ npx cordova plugin add https://github.com/phonegap/phonegap-plugin-contentsync.git Warning: using prerelease version 10.0.0-nightly.2020.7.14.d69f1084 (cordova-lib@10.0.0-nightly.2020.7.14.784f6b4b) Failed to fetch plugin https://github.com/phonegap/phonegap-plugin-contentsync.git via registry. Probably this is either a connection problem, or plugin spec is incorrect. Check your connection and plugin name/version/URL. CordovaError: CordovaError: Could not determine package name from output: phonegap-plugin-contentsync@1.4.2 node_modules/phonegap-plugin-contentsync ```
shankari commented 4 years ago

Aha! While creating the MCVE, figured out the issue. I have both cordova and phonegap as dependencies. I was working on upgrading cordova first, and had not yet started upgrading phonegap, which was still at 7.1.1. After upgrading phonegap to 9.0.0+cordova.9.0.0, the error disappeared. I guess 7.1.1 was close enough to 8.0.0 to work, but was too far from 9.0.0