Failed publishes of all builds

[rvagg]

@nodejs/build I might need some help investigating this I think, there's not enough of me to spend the time needed on this but Snap users are missing the latest releases.

All releases today were rejected by Snapcraft:

  found errors in file output: unusual mode 'rwxr-sr-x' for entry './bin', unusual mode 'rwxr-sr-x' for entry './lib'

Even master ("nodeedge") started failing today with the same problem, but it hasn't before.

I don't know if this is something new we've introduced across all branches, whether it's something new Launchpad is doing with the builds, or some new checking that Snapcraft has introduced.

The s is pretty weird; why is setgid on for these? Not unreasonable to have these rejected.

[pvarga-dni]

This does mean that currently, insecure versions of Node.js are installed by snap. For example, the default channel installs 16.14.0, however, the 16.14.2 security patch release has been out for more than a month.

[Trott]

I don't know if this is something new we've introduced across all branches, whether it's something new Launchpad is doing with the builds, or some new checking that Snapcraft has introduced.

@nodejs/releasers

[jdstrand]

bin and lib are presumably dirs and something is setting the sgid bit on them. This is an old check in Launchpad, so something must've changed. It's easy enough conceptually to fix in snapcraft.yaml, but it would be good to know why this is happening. Where is the snapcraft.yaml and where is a url for a failed build?

[rvagg]

@jdstrand it's in the root of this repo, it's generated from https://github.com/nodejs/snap/blob/master/snapcraft.yaml.sh and is done for each branch in this repo that maps to an active release line (master maps to edge releases, node17 to Node.js 17 etc., the logic is in the script for that).

Failed builds fill my inbox .. https://launchpad.net/~openjs/node-snap/+snap/node16/+build/1743840 is one of the latest. You should be able to find them via https://launchpad.net/~openjs/+snaps

Any help would be appreciated!

[richardlau]

@mhdawson and I spent a bit of time looking at this while looking into https://github.com/nodejs/snap/pull/35. I think the culprit is https://github.com/nodejs/snap/blob/c069b76642d3e1c33f5ab48abc37429a3ff8c29b/snapcraft.yaml.sh#L104-L112. More specifically it looks like the setgid bit is set when the yarn tarball is extracted with permissions:

$ wget https://yarnpkg.com/latest.tar.gz
--2022-04-28 12:56:23--  https://yarnpkg.com/latest.tar.gz
Resolving yarnpkg.com (yarnpkg.com)... 2606:4700::6810:ab63, 2606:4700::6812:7e64, 104.16.171.99, ...
Connecting to yarnpkg.com (yarnpkg.com)|2606:4700::6810:ab63|:443... failed: Network is unreachable.
Connecting to yarnpkg.com (yarnpkg.com)|2606:4700::6812:7e64|:443... failed: Network is unreachable.
Connecting to yarnpkg.com (yarnpkg.com)|104.16.171.99|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://classic.yarnpkg.com/latest.tar.gz [following]
--2022-04-28 12:56:23--  https://classic.yarnpkg.com/latest.tar.gz
Resolving classic.yarnpkg.com (classic.yarnpkg.com)... 2600:1f1c:471:9d01:205d:4ea3:d46b:9cc6, 2600:1f1c:471:9d00:64a9:5908:2245:64e0, 54.241.68.193, ...
Connecting to classic.yarnpkg.com (classic.yarnpkg.com)|2600:1f1c:471:9d01:205d:4ea3:d46b:9cc6|:443... failed: Network is unreachable.
Connecting to classic.yarnpkg.com (classic.yarnpkg.com)|2600:1f1c:471:9d00:64a9:5908:2245:64e0|:443... failed: Network is unreachable.
Connecting to classic.yarnpkg.com (classic.yarnpkg.com)|54.241.68.193|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/yarnpkg/yarn/releases/download/v1.22.18/yarn-v1.22.18.tar.gz [following]
--2022-04-28 12:56:23--  https://github.com/yarnpkg/yarn/releases/download/v1.22.18/yarn-v1.22.18.tar.gz
Resolving github.com (github.com)... 140.82.112.3
Connecting to github.com (github.com)|140.82.112.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/4fd903b0-3535-4302-843d-a026d600b30d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T170156Z&X-Amz-Expires=300&X-Amz-Signature=2e4c0092fc8af7fbe3e2c0846717dce465c0e19cf6f1a33efc0de642a9235f5b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.18.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-04-28 12:56:24--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/4fd903b0-3535-4302-843d-a026d600b30d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T170156Z&X-Amz-Expires=300&X-Amz-Signature=2e4c0092fc8af7fbe3e2c0846717dce465c0e19cf6f1a33efc0de642a9235f5b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.18.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.108.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1246681 (1.2M) [application/octet-stream]
Saving to: ‘latest.tar.gz’

latest.tar.gz                          100%[============================================================================>]   1.19M  --.-KB/s    in 0.04s

2022-04-28 12:56:24 (32.1 MB/s) - ‘latest.tar.gz’ saved [1246681/1246681]

$ tar xpfz latest.tar.gz
$ ls -al
total 1220
drwxrwxr-x. 3 rlau rlau      48 Apr 28 13:11 .
drwxrwxrwt. 6 root root     235 Apr 28 12:56 ..
-rw-rw-r--. 1 rlau rlau 1246681 Mar 16 18:29 latest.tar.gz
drwxrwxr-x. 4 rlau rlau     101 Apr 28 13:11 yarn-v1.22.18
[rlau@drx-hemera snap]$ ls -al yarn-v1.22.18/
total 16
drwxrwxr-x. 4 rlau rlau  101 Apr 28 13:11 .
drwxrwxr-x. 3 rlau rlau   48 Apr 28 13:11 ..
-rw-r--r--. 1 rlau rlau 1355 Mar 16 14:29 LICENSE
-rw-r--r--. 1 rlau rlau 3353 Mar 16 14:29 README.md
drwxr-sr-x. 2 rlau rlau   83 Mar 16 14:29 bin
drwxr-sr-x. 2 rlau rlau   47 Mar 16 14:29 lib
-rw-r--r--. 1 rlau rlau  634 Mar 16 14:29 package.json
-rw-r--r--. 1 rlau rlau 2342 Mar 16 14:29 preinstall.js
$

[richardlau]

Even more specifically it appears to be the yarn-1.22.18 tarball:

$ ls -al
total 0
drwxrwxr-x. 2 rlau rlau   6 Apr 28 14:21 .
drwxrwxrwt. 6 root root 235 Apr 28 14:20 ..
$ wget https://github.com/yarnpkg/yarn/releases/download/v1.22.17/yarn-v1.22.17.tar.gz
--2022-04-28 14:21:12--  https://github.com/yarnpkg/yarn/releases/download/v1.22.17/yarn-v1.22.17.tar.gz
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/9b940159-ba7f-49ac-a9d6-f989acd4d92e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T182539Z&X-Amz-Expires=300&X-Amz-Signature=7ab0548806e43873dbdda1edadc03dcbd054afe0a1f776e0921431e7c5b902ac&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.17.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-04-28 14:21:12--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/9b940159-ba7f-49ac-a9d6-f989acd4d92e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T182539Z&X-Amz-Expires=300&X-Amz-Signature=7ab0548806e43873dbdda1edadc03dcbd054afe0a1f776e0921431e7c5b902ac&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.17.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1246198 (1.2M) [application/octet-stream]
Saving to: ‘yarn-v1.22.17.tar.gz’

yarn-v1.22.17.tar.gz                   100%[============================================================================>]   1.19M  --.-KB/s    in 0.04s

2022-04-28 14:21:12 (32.0 MB/s) - ‘yarn-v1.22.17.tar.gz’ saved [1246198/1246198]

$ wget https://github.com/yarnpkg/yarn/releases/download/v1.22.18/yarn-v1.22.18.tar.gz
--2022-04-28 14:21:20--  https://github.com/yarnpkg/yarn/releases/download/v1.22.18/yarn-v1.22.18.tar.gz
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/4fd903b0-3535-4302-843d-a026d600b30d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T182616Z&X-Amz-Expires=300&X-Amz-Signature=5c4f4f3217a465f6d95e719201dbdaba3780057f94f2767c5739c0193260662b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.18.tar.gz&response-content-type=application%2Foctet-stream [following]
--2022-04-28 14:21:20--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/49970642/4fd903b0-3535-4302-843d-a026d600b30d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220428%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220428T182616Z&X-Amz-Expires=300&X-Amz-Signature=5c4f4f3217a465f6d95e719201dbdaba3780057f94f2767c5739c0193260662b&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=49970642&response-content-disposition=attachment%3B%20filename%3Dyarn-v1.22.18.tar.gz&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1246681 (1.2M) [application/octet-stream]
Saving to: ‘yarn-v1.22.18.tar.gz’

yarn-v1.22.18.tar.gz                   100%[============================================================================>]   1.19M  --.-KB/s    in 0.04s

2022-04-28 14:21:20 (32.1 MB/s) - ‘yarn-v1.22.18.tar.gz’ saved [1246681/1246681]

$ tar xpf yarn-v1.22.17.tar.gz
$ tar xpf yarn-v1.22.18.tar.gz
$ ls -al yarn-v1.22.1*
-rw-rw-r--. 1 rlau rlau 1246198 Dec  7 23:09 yarn-v1.22.17.tar.gz
-rw-rw-r--. 1 rlau rlau 1246681 Mar 16 18:29 yarn-v1.22.18.tar.gz

yarn-v1.22.17:
total 16
drwxrwxr-x. 4 rlau rlau  101 Apr 28 14:21 .
drwxrwxr-x. 4 rlau rlau  104 Apr 28 14:21 ..
-rw-r--r--. 1 rlau rlau 1355 Oct 16  2021 LICENSE
-rw-r--r--. 1 rlau rlau 3353 Oct 16  2021 README.md
drwxr-xr-x. 2 rlau rlau   83 Oct 16  2021 bin
drwxr-xr-x. 2 rlau rlau   47 Oct 16  2021 lib
-rw-r--r--. 1 rlau rlau  598 Oct 16  2021 package.json
-rw-r--r--. 1 rlau rlau 2342 Oct 16  2021 preinstall.js

yarn-v1.22.18:
total 16
drwxrwxr-x. 4 rlau rlau  101 Apr 28 14:21 .
drwxrwxr-x. 4 rlau rlau  104 Apr 28 14:21 ..
-rw-r--r--. 1 rlau rlau 1355 Mar 16 14:29 LICENSE
-rw-r--r--. 1 rlau rlau 3353 Mar 16 14:29 README.md
drwxr-sr-x. 2 rlau rlau   83 Mar 16 14:29 bin
drwxr-sr-x. 2 rlau rlau   47 Mar 16 14:29 lib
-rw-r--r--. 1 rlau rlau  634 Mar 16 14:29 package.json
-rw-r--r--. 1 rlau rlau 2342 Mar 16 14:29 preinstall.js
$

yarn-1.22.18 was published on 15 March 2022, which is around the time we started seeing the failures.

[richardlau]

@arcanis Was the permissions change in the yarn tarball between 1.22.17 and 1.22.18 intentional?

[arcanis]

No - the 1.x automated deploy pipeline isn't feeling well at the moment so I made the release from my laptop, it may have been the cause. However I don't repro this locally 🤔

❯ typanion git:(main) ❯ curl -s https://registry.yarnpkg.com/yarn/-/yarn-1.22.18.tgz | tar tvz

-rw-r--r-- 0/0            1355 1985-10-26 09:15 package/LICENSE
-rwxr-xr-x 0/0            1025 1985-10-26 09:15 package/bin/yarn
-rwxr-xr-x 0/0              42 1985-10-26 09:15 package/bin/yarnpkg
-rwxr-xr-x 0/0              34 1985-10-26 09:15 package/bin/yarn.cmd
-rwxr-xr-x 0/0              30 1985-10-26 09:15 package/bin/yarnpkg.cmd
-rwxr-xr-x 0/0         5306316 1985-10-26 09:15 package/lib/cli.js
-rw-r--r-- 0/0            2342 1985-10-26 09:15 package/preinstall.js
-rw-r--r-- 0/0            9910 1985-10-26 09:15 package/lib/v8-compile-cache.js
-rwxr-xr-x 0/0            1015 1985-10-26 09:15 package/bin/yarn.js
-rw-r--r-- 0/0             634 1985-10-26 09:15 package/package.json
-rw-r--r-- 0/0            3353 1985-10-26 09:15 package/README.md

[arcanis]

Ah, I see you seem to be pulling the tarball directly from GitHub, which should be handled by @daniel15's build server, he may know more about this.

[richardlau]

@arcanis You need to pass the p flag to tar to restore permissions -- I'm guessing this is what snapcraft are doing behind the scenes.

[richardlau]

Although I think you're on to something and this might be specific to the GH tarball:

$ curl -sL https://github.com/yarnpkg/yarn/releases/download/v1.22.18/yarn-v1.22.18.tar.gz | tar tvz
drwxr-sr-x daniel/daniel     0 2022-03-16 14:29 yarn-v1.22.18/bin/
-rwxr-xr-x daniel/daniel  1015 2022-03-16 14:29 yarn-v1.22.18/bin/yarn.js
-rwxr-xr-x daniel/daniel    34 2022-03-16 14:29 yarn-v1.22.18/bin/yarn.cmd
-rwxr-xr-x daniel/daniel    42 2022-03-16 14:29 yarn-v1.22.18/bin/yarnpkg
-rwxr-xr-x daniel/daniel  1025 2022-03-16 14:29 yarn-v1.22.18/bin/yarn
-rwxr-xr-x daniel/daniel    30 2022-03-16 14:29 yarn-v1.22.18/bin/yarnpkg.cmd
drwxr-sr-x daniel/daniel     0 2022-03-16 14:29 yarn-v1.22.18/lib/
-rw-r--r-- daniel/daniel  9910 2022-03-16 14:29 yarn-v1.22.18/lib/v8-compile-cache.js
-rwxr-xr-x daniel/daniel 5306316 2022-03-16 14:29 yarn-v1.22.18/lib/cli.js
-rw-r--r-- daniel/daniel    1355 2022-03-16 14:29 yarn-v1.22.18/LICENSE
-rw-r--r-- daniel/daniel     634 2022-03-16 14:29 yarn-v1.22.18/package.json
-rw-r--r-- daniel/daniel    2342 2022-03-16 14:29 yarn-v1.22.18/preinstall.js
-rw-r--r-- daniel/daniel    3353 2022-03-16 14:29 yarn-v1.22.18/README.md
$ curl -sL https://registry.yarnpkg.com/yarn/-/yarn-1.22.18.tgz | tar tvz
-rw-r--r-- 0/0            1355 1985-10-26 04:15 package/LICENSE
-rwxr-xr-x 0/0            1025 1985-10-26 04:15 package/bin/yarn
-rwxr-xr-x 0/0              42 1985-10-26 04:15 package/bin/yarnpkg
-rwxr-xr-x 0/0              34 1985-10-26 04:15 package/bin/yarn.cmd
-rwxr-xr-x 0/0              30 1985-10-26 04:15 package/bin/yarnpkg.cmd
-rwxr-xr-x 0/0         5306316 1985-10-26 04:15 package/lib/cli.js
-rw-r--r-- 0/0            2342 1985-10-26 04:15 package/preinstall.js
-rw-r--r-- 0/0            9910 1985-10-26 04:15 package/lib/v8-compile-cache.js
-rwxr-xr-x 0/0            1015 1985-10-26 04:15 package/bin/yarn.js
-rw-r--r-- 0/0             634 1985-10-26 04:15 package/package.json
-rw-r--r-- 0/0            3353 1985-10-26 04:15 package/README.md
$

[richardlau]

Also FWIW we're not pulling the GH tarball directly, we're pulling https://yarnpkg.com/latest.tar.gz https://github.com/nodejs/snap/blob/c069b76642d3e1c33f5ab48abc37429a3ff8c29b/snapcraft.yaml.sh#L106

[arcanis]

Yep exactly. I think the build server got moved, and the permissions of the build directory probably accidentally changed in the process. I can't unfortunately update the archive on GitHub directly, since the archive is provided with an .asc key I can't generate on my side 🤔

[jdstrand]

Based on all of this, it's clear that the yarn-1.22.18.tgz is the issue. snapcraft builds run as root and will unpack with -p which is why the sgid bit persists after unpacking. When the snap is converted to a squashfs, the permissions remain and then the review-tools in the snap store see the unusual permissions and complain.

IMHO, the best fix is for a new yarn build (eg yarn-1.22.19.tgz) to have the corrected permissions (which IMO is warranted anyway since it sounds like the tarball was built outside of a build server; the fact that there are sgid permissions in the upload suggests that perhaps (portions of) the build/release ran as root which as an outsider raises my eyebrows). In the meantime, the snapcraft.yaml fix in https://github.com/nodejs/snap/pull/36 would workaround the issue in the current yarn-1.22.18.tgz.

[Daniel15]

Ah, I see you seem to be pulling the tarball directly from GitHub, which should be handled by @Daniel15's build server, he may know more about this.

The tarball is normally built as part of the CircleCI build system, however there was some issue with it so we built it manually. Originally @arcanis built it on his laptop, and I had to manually re-run the build on one of my own systems to get the .deb etc artifacts too.

I didn't run the build as root. It looks like the files had the setgid flag set because on my build server I have the Yarn source code in /usr/local/src/yarn, which is owned by daniel but inherited the setgid flag that Debian sets on /usr/local/ by default (which it does to ensure the staff group can access all files in there). I don't usually build Yarn on this system, so I never noticed this.

@arcanis Is it possible to fix the CircleCI build so we can use its tarball instead?

since the archive is provided with an .asc key I can't generate on my side 🤔

I can provide you with the GPG signing key if you'd like.

[rvagg]

👌 nice detective work @richardlau

[rvagg]

releases rolling into the store now; there was a failure on master for amd64 and node12 for i386, looks like a transient download problem and I'm gong to ignore both of them because the rest seem to be going OK and those two are lower priority.