search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
18.32k stars 14.29k forks source link

Get rid of sha1 for fixed-output derivations in nixpkgs #77238

Open grahamc opened 4 years ago

grahamc commented 4 years ago

Issue description

We're in 2020 and:

Biggest usage of sha1 is files generated for Node packages:

2372    pkgs/development/node-packages/node-packages-v10.nix
1741    pkgs/servers/web-apps/codimd/yarn.nix
1644    pkgs/applications/version-management/gitlab/yarnPkgs.nix
1485    pkgs/servers/monitoring/prometheus/webui-yarndeps.nix
1481    pkgs/servers/gotify/yarndeps.nix
476 pkgs/development/tools/yarn2nix-moretea/yarn2nix/yarn.nix
421 pkgs/applications/networking/cluster/spacegun/node-packages.nix
290 pkgs/servers/rippled/package.nix
213 pkgs/development/compilers/elm/packages/node-packages.nix
189 pkgs/development/web/remarkjs/node-packages.nix
169 pkgs/development/mobile/androidenv/generated/packages.nix
141 pkgs/development/node-packages/node-packages-v12.nix
127 pkgs/applications/networking/instant-messengers/riot/riot-desktop-yarndeps.nix
116 pkgs/servers/matrix-synapse/matrix-appservice-slack/node-packages.nix
88  pkgs/misc/base16-builder/node-packages-generated.nix
74  pkgs/development/mobile/androidenv/generated/addons.nix
73  pkgs/development/node-packages/node-packages-v13.nix
70  pkgs/development/compilers/graalvm/default.nix
66  pkgs/applications/networking/instant-messengers/matrix-recorder/node-packages.nix
48  pkgs/tools/networking/airfield/node-packages.nix
48  pkgs/development/misc/google-clasp/node-packages.nix
36  pkgs/development/mobile/androidenv/generated/system-images-android.nix
33  pkgs/servers/web-apps/cryptpad/node-packages-generated.nix
33  pkgs/development/mobile/androidenv/generated/system-images-google_apis.nix
10  pkgs/development/mobile/androidenv/generated/system-images-android-tv.nix
6   pkgs/tools/package-management/nixui/node-packages.nix
6   pkgs/development/mobile/androidenv/generated/system-images-google_apis_playstore.nix
6   pkgs/development/mobile/androidenv/generated/system-images-android-wear.nix
6   pkgs/development/mobile/androidenv/convertaddons.xsl
4   pkgs/development/mobile/androidenv/generated/system-images-android-wear-cn.nix
2   pkgs/development/mobile/androidenv/convertpackages.xsl
2   pkgs/applications/networking/browsers/mozilla-plugins/google-talk-plugin/default.nix
1   pkgs/tools/typesetting/tex/texlive/default.nix
1   pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/generateNix.js
1   pkgs/development/tools/unity3d/default.nix
1   pkgs/development/mobile/androidenv/convertsystemimages.xsl
1   pkgs/development/libraries/wxsqliteplus/default.nix
1   pkgs/development/libraries/wxsqlite3/default.nix
1   pkgs/development/haskell-modules/configuration-hackage2nix.yaml
1   pkgs/development/haskell-modules/configuration-common.nix
1   pkgs/build-support/vm/rpm/rpm-closure.pl
1   pkgs/applications/office/grisbi/default.nix
1   pkgs/applications/graphics/gcolor2/default.nix
1   doc/release-notes.xml

Steps to resolve, higher level

Files to address

edolstra commented 4 years ago

We can script most of this by using maintainers/scripts/find-tarballs.nix to get all the URLs/hashes, fetching them and then doing s/$sha1/$sha256 on the Nix expression. Or use tarballs.nixos.org to map sha1 hashes to sha512:

$ curl -v https://tarballs.nixos.org/sha1/e410a52dcff3d5c6c3d448b68a026d04ccd744be
location: /sha512/90ae204071cdab3b5e0da28709afa8ac0a5ee13737254cded76a6b8cf323ed2b42390aaa10dbb7935b2aaa57504ceba21306a7eaeea088ab85ee9bdb0a05b530
cdepillabout commented 4 years ago

@grahamc Thanks for looking into this. I think these types of security related fixes are important!

I was wondering how you determined the list of files using sha1?

I wanted to fix the haskell-related files (pkgs/development/haskell-modules/configuration-common.nix and pkgs/development/haskell-modules/configuration-hackage2nix.yaml), so I grepped through the files looking for the strings sha1 and hash, but I couldn't find anything suspicious. Also configuration-hackage2nix.yaml doesn't appear to have any hashes at all.

grahamc commented 4 years ago

I did a simple search of sha1 =. It has a few false positives, for sure :) Please tick them as done if you find them!

7c6f434c commented 4 years ago

git grep -l -E '(^| )sha1\> *= *' should catch less false positives, I believe.

Once we clean up all the generators, we can rerun with \<sha1\> *= as a safety check.

kira-bruneau commented 4 years ago

All the packages in node-packages-v10.nix that use sha1, only do so because they don't list a sha512 hash in the npm registry. Regenerating this file is already slow enough, if we wanted to use sha256 or sha512, node2nix (and yarn2nix) would also have to download the files.

For example, acorn 4.0.13 doesn't have a sha512 hash:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/4.0.13 | jq .dist'
{
  "shasum": "105495ae5361d697bd195c825192e1ad7f253787",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-4.0.13.tgz"
}

but version 7.1.0 does:

nix-shell -p curl jq --run 'curl http://registry.npmjs.org/acorn/7.1.0 | jq .dist'
{
  "integrity": "sha512-kL5CuoXA/dgxlBbVrflsflzQ3PAas7RYZB52NOm/6839iVYJgKMJ3cQJD+t2i5+qFa8h3MDpEOJiS64E8JLnSQ==",
  "shasum": "949d36f2c292535da602283586c2477c57eb2d6c",
  "tarball": "https://registry.npmjs.org/acorn/-/acorn-7.1.0.tgz",
  "fileCount": 11,
  "unpackedSize": 1104477,
  "npm-signature": "-----BEGIN PGP SIGNATURE-----\r\nVersion: OpenPGP.js v3.0.4\r\nComment: https://openpgpjs.org\r\n\r\nwsFcBAEBCAAQBQJdibjuCRA9TVsSAnZWagAAmQgQAIkJPo9i+wdBLLZY8yjT\nHI82RsWU5OvIQxhE6H9qLiEfF57ZSn3LfKVugsEocc84loCiJkkd7vsGsidJ\nRMbkEIK2ENjH1VcUHdHq8PnA/HgnMgHNq6nGg+h6aXLsN/H5SFdyvWedoLXw\n5bwhLbfURV/00bx2himrt/RBNL8SxfowdpQ/ps5+mAWUzLTf7D2+PchYktve\nRJ1lSwpshrgu4AxUdbRmqDEH7+Wm5SutB/EMkQPxQV/sJviFzB8qqFlQopJS\n83RuvIhQb7ZNUQeh5A1+pQocFU+yd227m9tAr69UeES8vinI+0FYVh5GNh0y\nk5MoiQiG3LROtO1RNea0uYGv8W7cU9CK/6MnHbDMUgr+fpEeU/VGItZfOLhB\nTg10PpYCKtQCsF+Z36YLTXgxeWEoRvvKWmJET/MDQzBM/pRzMRpPHiWn6E9T\nl9wt1WLix2qM1jymwaq+0H6hcCtu0ioY7qn4GGeCAfVlBxWYcydNR11Rb++n\nO3oXEqhc8oxKy+hIDHIMDDz/YfCrJ9q7ZzMO9Ie5dzOqeLxty4fbuDeLcD9U\nkbuA+I8dfEFmO2UtRwyx3k9yjX2Tsf2oAC8VOTINN8TXiauyTjWyU//lSKDX\nmUHBDXXZhKWgygi3TMNW7gvmzYuzmDcbkienQeoADvONpCuJNmraVaC33n1h\ncQmj\r\n=blZm\r\n-----END PGP SIGNATURE-----\r\n"
}
vcunat commented 4 years ago

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

worldofpeace commented 4 years ago

@grahamc I removed this as a blocker because I'm not convinced it's something we should block release on. Thoughts?

prusnak commented 4 years ago
prusnak commented 4 years ago
prusnak commented 4 years ago
prusnak commented 4 years ago
xworld21 commented 3 years ago

texlive: it has one "sha1 =" string but the code is generic – pkgs/tools/typesetting/tex/texlive/fixedHashes.nix has over 8k sha1 hashes. I haven't looked into complications when generating these, but if I look right, the file's size would increase roughly 460k -> 624k. (Seems acceptable, I suppose, given the circumstances.)

@vcunat (edit: I should be asking @veprbl) I can do the sha1 -> sha256 conversion for texlive. Now the increase is 498k -> 676k. Just give a thumbs up if you'd like a PR (or a thumbs down!).

vcunat commented 3 years ago

Well, I don't maintain texlive anymore, for years already. But I assume that what I wrote above still holds (approximately).

Stunkymonkey commented 3 years ago

i think the remaining sha1-packages are all related to node2nix or yarn2nix.

jtojnar commented 2 years ago

yarn2nix will now use the integrity field from yarn.lock when present (available from at least 2018), which is typically sha512 for packages ¿uploaded to npm in the last X years? – for example, yarn.lock of a random modern project contains 319 sha1 and 1624 sha512 entries.

We probably still want to port the TOFU script https://github.com/NixOS/nixpkgs/blob/bb9bd465b625bfc971908c5d3d84ce517e1c0691/pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/fixPkgAddMissingSha1.js to a different hash schema and maybe even switch to TOFU for packages using sha1 in integrity field.

prusnak commented 2 years ago

We probably still want to port the TOFU script https://github.com/NixOS/nixpkgs/blob/bb9bd465b625bfc971908c5d3d84ce517e1c0691/pkgs/development/tools/yarn2nix-moretea/yarn2nix/lib/fixPkgAddMissingSha1.js to a different hash schema

I sent a draft PR https://github.com/NixOS/nixpkgs/pull/149834

aikooo7 commented 9 months ago

Hello, I would like to help with this, any suggestions where I can get started?

nbraud commented 2 months ago

I went and updated the list in the issue, checking all files not yet marked as done (in some cases updating their path) and adding a few I found with rg -l -g '*.nix' -g '*.json' 'sha1 = "|"sha1": "|"sha1-' pkgs/.

PS: I looked through the remaining cases, and

The most sensible approach for androidenv and minecraft-servers might be to get the upstream to publish metadata with non-broken hashes. Do we have relevant contacts at Google and Mojang?