Install/update ends with npm error: "invalid version dev-master"

[mindplay-dk]

This project looks great, and I'm really excited about the prospect of finally being able to manage the front-end dependencies of our Composer packages!

As of yet, I haven't had any luck trying it out though - every composer install or composer update ends with an npm error.

Here's the (verbose) output:

$ composer install -v
KODUS 2 installer enabled
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Dependency resolution completed in 0.002 seconds
Analyzed 290 packages to resolve dependencies
Analyzed 1058 rules to resolve dependencies
Nothing to install or update
Package http-interop/http-server-handler is abandoned, you should avoid using it. Use psr/http-server-handler instead.
Package http-interop/http-server-middleware is abandoned, you should avoid using it. Use psr/http-server-middleware instead.
Generating autoload files
Merging Composer dependencies in the asset package
Installing npm dependencies
npm ERR! Invalid version: "dev-master"

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/mindplay/.npm/_logs/2018-09-06T09_16_38_957Z-debug.log
Fallback to previous state for the Asset package
Fallback to previous state for Composer
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Dependency resolution completed in 0.003 seconds
Analyzed 290 packages to resolve dependencies
Analyzed 1058 rules to resolve dependencies
Nothing to install or update
Package http-interop/http-server-handler is abandoned, you should avoid using it. Use psr/http-server-handler instead.
Package http-interop/http-server-middleware is abandoned, you should avoid using it. Use psr/http-server-middleware instead.
Generating autoload files

  [RuntimeException]
  The asset manager ended with an error

Exception trace:
 () at /mnt/c/workspace/test/staging_project/vendor/foxy/foxy/Solver/Solver.php:108
 Foxy\Solver\Solver->solve() at /mnt/c/workspace/test/staging_project/vendor/foxy/foxy/Foxy.php:131
 Foxy\Foxy->solveAssets() at n/a:n/a
 call_user_func() at phar:///usr/bin/composer/src/Composer/EventDispatcher/EventDispatcher.php:176
 Composer\EventDispatcher\EventDispatcher->doDispatch() at phar:///usr/bin/composer/src/Composer/EventDispatcher/EventDispatcher.php:96
 Composer\EventDispatcher\EventDispatcher->dispatchScript() at phar:///usr/bin/composer/src/Composer/Installer.php:323
 Composer\Installer->run() at phar:///usr/bin/composer/src/Composer/Command/InstallCommand.php:119
 Composer\Command\InstallCommand->execute() at phar:///usr/bin/composer/vendor/symfony/console/Command/Command.php:242
 Symfony\Component\Console\Command\Command->run() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:843
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:193
 Symfony\Component\Console\Application->doRun() at phar:///usr/bin/composer/src/Composer/Console/Application.php:251
 Composer\Console\Application->doRun() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at phar:///usr/bin/composer/src/Composer/Console/Application.php:100
 Composer\Console\Application->run() at phar:///usr/bin/composer/bin/composer:59
 require() at /usr/bin/composer:24

install [--prefer-source] [--prefer-dist] [--dry-run] [--dev] [--no-dev] [--no-custom-installers] [--no-autoloader] [--no-scripts] [--no-progress] [--no-suggest] [-v|vv|vvv|--verbose] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--ignore-platform-reqs] [--] [<packages>]...

Here's the output of the debug log:

0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node', '/usr/bin/npm', 'install' ]
2 info using npm@6.3.0
3 info using node@v9.11.2
4 verbose npm-session d6096af3659c374f
5 silly install runPreinstallTopLevelLifecycles
6 silly preinstall staging_project
7 info lifecycle @~preinstall: @
8 silly install loadCurrentTree
9 silly install readLocalPackageData
10 timing stage:loadCurrentTree Completed in 12ms
11 silly install loadIdealTree
12 silly install cloneCurrentTreeToIdealTree
13 timing stage:loadIdealTree:cloneCurrentTree Completed in 0ms
14 silly install loadShrinkwrap
15 timing stage:loadIdealTree:loadShrinkwrap Completed in 5ms
16 silly install loadAllDepsIntoIdealTree
17 silly fetchPackageMetaData error for @composer-asset/kodus--media@file:vendor/foxy/composer-asset/kodus/media Invalid version: "dev-master"
18 timing stage:rollbackFailedOptional Completed in 0ms
19 timing stage:runTopLevelLifecycles Completed in 251ms
20 silly saveTree staging_project
21 verbose stack Error: Invalid version: "dev-master"
21 verbose stack     at Object.fixVersionField (/usr/lib/node_modules/npm/node_modules/normalize-package-data/lib/fixer.js:191:13)
21 verbose stack     at /usr/lib/node_modules/npm/node_modules/normalize-package-data/lib/normalize.js:32:38
21 verbose stack     at Array.forEach (<anonymous>)
21 verbose stack     at normalize (/usr/lib/node_modules/npm/node_modules/normalize-package-data/lib/normalize.js:31:15)
21 verbose stack     at new Manifest (/usr/lib/node_modules/npm/node_modules/pacote/lib/finalize-manifest.js:121:3)
21 verbose stack     at tarballedProps.then.props (/usr/lib/node_modules/npm/node_modules/pacote/lib/finalize-manifest.js:50:13)
21 verbose stack     at tryCatcher (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
21 verbose stack     at Promise._settlePromiseFromHandler (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:512:31)
21 verbose stack     at Promise._settlePromise (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:569:18)
21 verbose stack     at Promise._settlePromiseCtx (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:606:10)
21 verbose stack     at Async._drainQueue (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:138:12)
21 verbose stack     at Async._drainQueues (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:143:10)
21 verbose stack     at Immediate.Async.drainQueues [as _onImmediate] (/usr/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:17:14)
21 verbose stack     at runCallback (timers.js:763:18)
21 verbose stack     at tryOnImmediate (timers.js:734:5)
21 verbose stack     at processImmediate (timers.js:716:5)
22 verbose cwd /mnt/c/workspace/test/staging_project
23 verbose Linux 4.4.0-17134-Microsoft
24 verbose argv "/usr/bin/node" "/usr/bin/npm" "install"
25 verbose node v9.11.2
26 verbose npm  v6.3.0
27 error Invalid version: "dev-master"
28 verbose exit [ 1, true ]

I am just trying it out, so for now, I've manually listed two packages that have package files via configuration in my project's composer.json:

    "config": {
        "foxy": {
            "enable-packages": {
                "kodus/media": true,
                "kodus/media-admin": true
            }
        }
    }

Both of these packages are on unable dev-branches, both named 1.0.0.

The package.json files in those two packages both list some "dependencies" for some npm packages.

If understand correctly, the idea is Foxy will aggregate these to a collective package.json for the project containing all the npm-dependencies of those two packages?

Any idea what I'm doing wrong?

[francoispluchino]

All asset dependencies must be defined in the the package.json file, whether in your project or in any of your PHP libraries. So, all Foxy config regard the PHP packages, and not the asset packages. About the config.foxy.enable-packages config, see the doc Enable/disable manually the PHP packages with an asset package definition to understand how this option works.

To understand how Foxy works, I suggest you read the questions from the FAQ and mainly: How does the plugin work? and Is Foxy useful if my asset dependencies are defined only in my project?.

About your 2 packages kodus/media and kodus/media-admin, are these PHP packages or JS packages?

[mindplay-dk]

All asset dependencies must be defined in the the package.json file, whether in your project or in any of your PHP libraries

That's what I've done.

About the config.foxy.enable-packages config, see the doc

Yes, quoting from the manual:

some public PHP package already uses the package.json file to handle their asset dependencies, but Foxy is not enabled for this package. In this case, you can manually enable the PHP packages to be scanned in your project.

That's what I tried, and it triggers the npm failure as described above.

Note that I have been able to install fgh151/core-assets to my project just for testing - it requires foxy/foxy, so Foxy picks up the package.json of this package correctly without needing to use the configuration setting in the project.

I just haven't had any luck installing a package via the configuration setting.

Best guess, from the console output, this has something to do with the fact I'm using an unstable Composer package on a dev-branch?

17 silly fetchPackageMetaData error for @composer-asset/kodus--media-admin@file:vendor/foxy/composer-asset/kodus/media-admin Invalid version: "1.0.0.x-dev"
18 timing stage:rollbackFailedOptional Completed in 1ms
19 timing stage:runTopLevelLifecycles Completed in 304ms
20 silly saveTree staging_project
21 verbose stack Error: Invalid version: "1.0.0.x-dev"
21 verbose stack     at Object.fixVersionField (/usr/lib/node_modules/npm/node_modules/normalize-package-data/lib/fixer.js:191:13)

The version number is correctly 1.0.0.x-dev in Composer terms, but perhaps this version number isn't written correctly to vendor/foxy/composer-asset/kodus/media-admin/package.json? (I can't tell, because the changes are rolled back after the command fails.)

About your 2 packages kodus/media and kodus/media-admin, are these PHP packages or JS packages?

These are Composer packages, both of which have a package.json in the root.

[francoispluchino]

Ok, I understand a little better, can you copy me your composer.json file from the library and its package.json file located in the folder ./vendor/foxy/composer-asset/<your-library>?

[francoispluchino]

Logically, the version used is not converted correctly (1.0.0.x-dev and not 1.0.0.0), but I just want to check.

[mindplay-dk]

Here's package.json from the kodus/media-admin Composer package:

{
  "devDependencies": {
    "css-loader": "^0.28.11",
    "mini-css-extract-plugin": "^0.4.0",
    "node-sass": "^4.9.0",
    "sass-loader": "^7.0.1",
    "source-map-loader": "^0.2.3",
    "ts-loader": "^4.3.0",
    "typescript": "^2.9.2",
    "webpack": "^4.10.1",
    "webpack-cli": "^2.1.4"
  },
  "dependencies": {
    "preact": "^8.2.9"
  },
  "scripts": {
    "build": "npm run build-webpack && npm run build-dts",
    "build-webpack": "webpack --mode production",
    "build-dts": "tsc --project typescript/tsconfig.dts.json --outFile assets/media.d.ts",
    "watch": "webpack --mode development --watch"
  }
}

And here's the vendor/foxy/composer-asset/kodus/media-admin/package.json file:

{
    "devDependencies": {
        "css-loader": "^0.28.11",
        "mini-css-extract-plugin": "^0.4.0",
        "node-sass": "^4.9.0",
        "sass-loader": "^7.0.1",
        "source-map-loader": "^0.2.3",
        "ts-loader": "^4.3.0",
        "typescript": "^2.9.2",
        "webpack": "^4.10.1",
        "webpack-cli": "^2.1.4"
    },
    "dependencies": {
        "preact": "^8.2.9"
    },
    "scripts": {
        "build": "npm run build-webpack && npm run build-dts",
        "build-webpack": "webpack --mode production",
        "build-dts": "tsc --project typescript/tsconfig.dts.json --outFile assets/media.d.ts",
        "watch": "webpack --mode development --watch"
    },
    "name": "@composer-asset/kodus--media-admin",
    "version": "1.0.0.x-dev"
}

It's getting a literal copy of the Composer version number 1.0.0.x-dev, right?

Here's the generated project-level package.json as well:

{
    "private": true,
    "dependencies": {
        "@composer-asset/kodus--media-admin": "file:./vendor/foxy/composer-asset/kodus/media-admin"
    }
}

If I try to manually run npm install, I see the same problem:

$ npm install
npm ERR! Invalid version: "1.0.0.x-dev"

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/mindplay/.npm/_logs/2018-09-06T12_31_36_516Z-debug.log

Could this have something to do with node-semver having support for x-ranges? I don't think Composer has that, so maybe this is where the hangup is?

[francoispluchino]

Normally, there is a version conversion, so it's a bug at this level.

[francoispluchino]

I may have gone a little quick in my deduction ... The conversion of version only occurs if the version is not defined in the package.json file, otherwise, you must follow the Semver format used by NPM.

Did you manually enter the version in the package.json file?

[mindplay-dk]

Did you manually enter the version in the package.json file?

I did not. (You can see the package.json I posted above.)

Version numbers exclusively come from git-tags/branches in all our projects and packages.

[mindplay-dk]

I traced through AssetUtil::formatPackage() and the version key isn't set when it enters the function.

The first condition if (!isset($packageValue['version'])) evaluates true.

The second condition if (0 === strpos($version, 'dev-') && isset($extra['branch-alias'][$version])) evaluates false.

The individual conditions 0 === strpos($version, 'dev-') and isset($extra['branch-alias'][$version]) both evaluate false.

[francoispluchino]

Can you try with the ^1.0.0@dev version of Foxy?

[mindplay-dk]

It's now writing a version number 1.0.0.0, but...

$ npm install
npm ERR! Invalid version: "1.0.0.0"

[francoispluchino]

That's what it seemed to me, I just wanted to be sure, so I just limited the converted version to 3 levels with 5b7cb97abc07fbff4eec87abd941be5c6a532c54.

[mindplay-dk]

Okay, it installs now :-)

But simply erasing the rest of the version number... won't this create a problem when the package goes from an unstable dev-branch to a tagged release? The version number will be identical then - will npm still update the dependency if the version number hasn't changed?

(It's a bit of an edge-case maybe, but could lead to annoying support questions and silent bugs.)

[francoispluchino]

Semver doesn't support 4 levels in the version, but only 3. Normally this should not be a problem, just like versions with 3 levels in development.

[mindplay-dk]

Yes, but it does support pre-release versions such as 1.0.0-x.7.z.92 - I think the problem here is that node-semver inteprets the x notation used by Composer in version numbers as x-ranges when they occur in version constraints.

I'm not sure you can simply copy a version number and to a version constraint and expect it work? They have different syntax? I don't have a suggestion for how to solve it though.

[francoispluchino]

Normally, it is better to use the format of each tool, so add the version in the package.json file of your library. Foxy do only a fallback of version, because the version is required by NPM, but we could create a fallback more efficient in the same genre as Composer Asset Plugin, but in the opposite direction (see the SemverConverter class). But this is another issue in this case.

[mindplay-dk]

Normally, it is better to use the format of each tool, so add the version in the package.json file of your library

Right, very good point - we should think of the npm-package inside the Composer package as an actual package and version it.

Perhaps even consider adding a warning on the console when the version number is missing? As you say, defaulting to the Composer version is only a fallback - versioning the npm-package explicitly is much safer in terms of correctly versioning breaking/non-breaking changes to any client-side APIs exposed by the npm-package; default to the Composer version will effectiely just "randomly" version the built-in npm-package, and it's probably a very bad idea to rely on this.

Maybe this should be documented at least?

[francoispluchino]

Yes, why not! If you are motivated to add this section in the doc, I will merge your PR with pleasure :-).