npm-shrinkwrap github URLs not supported by Yocto bitbake npmsw handler

pdcastro commented 3 years ago

Users trying to build the CLI in a Yocto image reported errors regarding the github style URLs in the CLI's npm-shrinkwrap.json file:

https://forums.balena.io/t/balena-cli-compilation-issues-in-yocto/252703

github:balena-io-modules/unbzip2-stream#942fc218013c14adab01cf693b0500cf6ac83193", needs to be modified to: "git+https://github.com/resin-io-modules/unbzip2-stream.git#942fc218013c14adab01cf693b0500cf6ac83193",

The problem is that bitbake has a custom npmsw handler for these files and it doesn't support things like the github: prefix or from: repos that point to branches not merged to master. Check out the source here: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npmsw.py?id=507a47a4e5077d5f8f76d9629be6b871dfd8eb90#n125

So the best workaround for now may be the following in your yocto copy of npm-shrinkwrap.json:

replace github: prefixes with git+https:

for modules using commits from a branch not merged to master, append ;branch=foobar between the url and the commit hash and avoid changing the commit itself

eg.
git+https://github.com/balena-io-modules/bonjour.git;branch=fixed-mdns#e018851dc823b4b3f670f658f71d0c1c7f3e637c
git+https://github.com/balena-io-modules/multicast-dns.git;branch=listen-on-all-interfaces#a15c63464eb43e8925b187ed5cb9de6892e8aacc
git+https://github.com/balena-io-modules/resin-discoverable-services.git;branch=find-on-all-interfaces#afca9e4700ec5ef82aa897f14bd5a46f06518061
git+https://github.com/balena-io-modules/unbzip2-stream.git;branch=core-streams#942fc218013c14adab01cf693b0500cf6ac83193

Github URL formats supported by the standard npm tool are documented at: https://docs.npmjs.com/cli/v6/commands/npm-install

Consider making the URLs in the CLI's npm-shrinkwrap.json file "conform to the older standard" in order to be compatible with bitbake / Yocto.

jellyfish-bot commented 3 years ago

[pdcastro] This issue has attached support thread https://jel.ly.fish/e21343ce-0ad7-4bf2-8dc3-fc1fbcc809b0

pdcastro commented 3 years ago

@klutchell also reported that npmsw does not support "the from: line to specify branches with commits not merged to master (branch must be specified in the version: line before the commit hash)".

I think an example would be:

"multicast-dns": {
  "version": "git+https://github.com/resin-io-modules/multicast-dns.git#a15c63464eb43e8925b187ed5cb9de6892e8aacc",
  "from": "git+https://github.com/resin-io-modules/multicast-dns.git#listen-on-all-interfaces",
  ...
},

klutchell commented 3 years ago

@pdcastro correct, the workaround is to inject ;branch= into the version: string since as far as I can tell the from: string is completely ignored and it looks for the commit in the primary branch.

"multicast-dns": {
  "version": "git+https://github.com/resin-io-modules/multicast-dns.git;branch=listen-on-all-interfaces#a15c63464eb43e8925b187ed5cb9de6892e8aacc",
  "from": "git+https://github.com/resin-io-modules/multicast-dns.git#listen-on-all-interfaces",
  ...
},

This may break a normal npm install since the ;branch= syntax is from bitbake. Another reason we probably don't want to support these changes and would be better to patch bitbake upstream.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14236

balena-io / balena-cli

npm-shrinkwrap github URLs not supported by Yocto bitbake npmsw handler #2192