Open domenkozar opened 6 years ago
sqlite> .mode line
sqlite> select * from NARs where url = 'nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6';
cache = 2
storePath = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
url = nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
compression = xz
fileHash =
fileSize =
narHash = sha256:0yifcx0975amb366q6nshb5qm2rrppy2f7h7k4q66hlblxr8m3rp
narSize = 4655744
refs = 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
deriver = 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
signedBy = hydra.iohk.io
timestamp = 1514632193
$ curl https://hydra.iohk.io/0p7x4l2zab62bghdhc5ssbqi7kpwnblj.narinfo
StorePath: /nix/store/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
URL: nar/0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6
Compression: xz
NarHash: sha256:0qsf2vki8g6p3lhn1jqn1vpq8s4lqanjmnfqmg88m9pc7ycgbjmm
NarSize: 4655736
References: 0p7x4l2zab62bghdhc5ssbqi7kpwnblj-deriving-compat-0.3.6 6bw3rq0xhayyj9dl158rqwi8ap395aa0-transformers-compat-0.5.1.4 73n08rwpzjqwvyizwphybcjfd85pkvqh-ghc-8.0.2 fjn0g0q5wrphysg3ijydijcpi274nwi8-gmp-6.1.2 l48biijfr1j6d5kdg911051x2phfjrz7-glibc-2.25
Deriver: 8f35pf501k765mdz6vjwg1pibmgxckwp-deriving-compat-0.3.6.drv
System: x86_64-linux
Sig: hydra.iohk.io:fNL1+eP/1CVem0xispVMChbktbEjujeAI0Eu4Te8ka21GRg2zUu+lKnp30Eb6lQjiqC83YVc7DEkEJ5M26B9BQ==
lesson learned: don't GC your binary cache
Or probably introduce a config for reducing the disk cache ttl for both positive/negative lookups. Which should be useful for binary caches that are GCed.
https://github.com/NixOS/nix/blob/1.11.15/scripts/download-from-binary-cache.pl.in
i think it can only expire negative hits, so positives stick around forever
nix 2.0 will expire things better
So https://github.com/NixOS/nix/blob/c0015e87af70f539f24d2aa2bc224a9d8b84276b/src/libstore/nar-info-disk-cache.cc#L52 needs a config setting to tweak it.
There is another fix which doesn't require rm /nix/var/nix/binary-cache-v3.sqlite
. You can instead add --fallback
to your nix-build
command or (better) add --option build-fallback true
. The latter is more robust because it works for import-from-derivation, too.
See this other issue, which is related to this one: https://github.com/NixOS/nix/issues/1148
@domenkozar We just hit this in our system as well. Was there some activity on this issue since last year? Or is the solution still to not expire things on the cache server?
See narinfo-cache-positive-ttl
in man nix.conf
Looks like a duplicate of https://github.com/NixOS/nix/issues/2035 - or the other way around I guess
I really think the correct fix is for binary cache to advertise this setting and user min(user_setting, cache_setting)
.
Can't we just invalidate the entry when it turns out to be unusable and retry the substitution?
That's what @volth suggested in https://github.com/NixOS/nix/issues/2035#issuecomment-37788632 and it appears to solve this issue too.
Sure, that would be most reliable. Note that in IFD we'd have to make sure it works too.
Why do you think IFD is different? Afaik, it builds and substitutes normally.
I don't know the code base, I want to be sure IFD is tested.
It seems like this would be easily added at https://github.com/NixOS/nix/blob/master/src/libstore/binary-cache-store.cc#L89 to fetch narinfo without using the cache and getting the nar file again.
I've stumbled upon the same error message today. The original tar I depend on is https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz
, when I run:
nix-prefetch-url --unpack https://github.com/NixOS/nixpkgs-channels/archive/6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz
unpacking...
[18.5 MiB DL]
path is '/nix/store/wcmdxyqq4m2761aqyrhz1v28mj5a0wdj-6e1393654240cca71efe1c7712c870aad1ab0792.tar.gz'
0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr
But when instantiating a nix-shell, I get the error:
% nix-shell
unpacking 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz'...
error: hash mismatch in file downloaded from 'https://github.com/NixOS/nixpkgs-channels/archive/f9eba87bf03318587df8356a933f20cfbc81c6ee.tar.gz':
wanted: sha256:0vq5z3pmz5vf4637fhyg4ina3nhg29l94pnf962nw0b5lvwn9mxr
got: sha256:15cfqd3hargafkhfq5p9hw75dibnbajh06ha69k18scg9l1p1hnl
(use '--show-trace' to show detailed location information)
how can I keep the same references without breaking checksums in the future?
I've tried to reproduce this in https://github.com/NixOS/nix/pull/3969 but it seems to just work? Maybe my test is not testing the right thing?
I've found the problem, Nix will say it's copying it, but it will actually fallback to building it.
not sure if it helps, but I've just discovered this one:
building '/nix/store/j8z3r8nzig402vwdqww3f2a6bd05zc3q-source.drv'...
trying https://github.com/hashicorp/packer/archive/v1.6.2.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 124 100 124 0 0 558 0 --:--:-- --:--:-- --:--:-- 558
100 16.5M 0 16.5M 0 0 3803k 0 --:--:-- 0:00:04 --:--:-- 4288k
unpacking source archive /private/var/folders/7s/_k9r0mwd15b4pr4h78v5x2km0000gn/T/nix-build-source.drv-0/v1.6.2.tar.gz
hash mismatch in fixed-output derivation '/nix/store/1x9w8wf4qfgf2xnmafjmqd2mshzk47hr-source':
wanted: sha256:104jw2jcshzds74d7m4yqn6mbk7lgps6qnqmp6h395b1mdyjhink
got: sha256:0kr9whv3s0f8866yjmwg311j3kcj29bp5xwpnv43ama4m1mq3bm7
cannot build derivation '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv': 1 dependencies couldn't be built
error: build of '/nix/store/f8idr16gzbh6nbh546ym1d2qyiyi3907-packer-1.6.2.drv', '/nix/store/wp749d94mr8zx1cpjm388k9ggnbc2aay-ghc-8.10.2-with-packages.drv' failed
MacOS, the pinned nixpgs are:
builtins.fetchTarball {
name = "nixpkgs-unstable-2020-09-01";
url = "https://github.com/NixOS/nixpkgs-channels/archive/780da9407e6da081b505f299681c78833357ebe0.tar.gz";
sha256 = "0l78bgsil1h5i390d34x5qh3raji0wdxqhsqbqs3ihyi6i6c2403";
};
but then it works well with a more recent https://github.com/NixOS/nixpkgs-channels/commit/a0b3103d81bcd5049dc51598314fc437f6f317d1
builtins.fetchTarball {
name = "nixpkgs-unstable-2020-09-01";
url = "https://github.com/NixOS/nixpkgs-channels/archive/a0b3103d81bcd5049dc51598314fc437f6f317d1.tar.gz";
# Hash obtained using `nix-prefetch-url --unpack <url>`
sha256 = "0fcsc4qkqd3m4m24wyypppqiv5z06b6wsvsfa90x4qfzbadgdc0x";
}
Both checkouts were marked as "verifed" on github.
Maybe my reasoning is not right and it is possible to have hash mismatches for some packages in nixpkgs-unstable
? Does CI make sure that the entire package set is consistent with their checksums?
@coderfromhere @avanov The issues you're seeing seem to be distinct from the original issue discussed here, because your mismatches occur in fetchers and fixed-output derivations, not during substitution.
These are usually caused by changes to the contents backing those URLs you're fetching from. For example, maintainers can change git tags or re-upload an "improved" release tarball and cause a hash mismatch that way. The reason it works on the recent commit, is because it was fixed four days ago.
@coderfromhere In your case, a prefetching command should be able to produce the correct hash. Please make sure that you're using the same parameters in your command and expression and that you're using a fixed-output derivation, not an evaluation-time fetcher like fetchurl
. If the issue remains, please open a separate issue.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/hash-mismatch-in-downloaded-path/11056/1
The minimum actionable here is to improve the error message.
I marked this as stale due to inactivity. → More info
I honestly wonder if this is the improved error message:
darwin-rebuild switch
building the system configuration...
error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5':
specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1
got: sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1
(use '--show-trace' to show detailed location information)
I tried rm $HOME/.cache/nix/binary-cache-v*.sqlite*
as well as narinfo-cache-negative-ttl = 0
without success.
Don't know what to do about it.
I honestly wonder if this is the improved error message:
darwin-rebuild switch building the system configuration... error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5': specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1 got: sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1 (use '--show-trace' to show detailed location information)
I tried
rm $HOME/.cache/nix/binary-cache-v*.sqlite*
as well asnarinfo-cache-negative-ttl = 0
without success.Don't know what to do about it.
This might be https://github.com/NixOS/nixpkgs/issues/179272. For the URL in the log, I get a the "specified:" hash on (case-sensitive) linux, and the "got:" hash on (case-insensitive) macOS. Was the "specified:" hash computed on a case-sensitive filesystem?
I honestly wonder if this is the improved error message:
darwin-rebuild switch building the system configuration... error: hash mismatch in file downloaded from 'https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5': specified: sha256:1z28a3gqbv62sxahlssc5a722kh46f26f5ss3arbxpv7a1272vf1 got: sha256:1n2z95321nccy3y9h8zw3gsgzwkaigypkybqjva31qd6xb4dl3d1 (use '--show-trace' to show detailed location information)
I tried
rm $HOME/.cache/nix/binary-cache-v*.sqlite*
as well asnarinfo-cache-negative-ttl = 0
without success. Don't know what to do about it.This might be NixOS/nixpkgs#179272. For the URL in the log, I get a the "specified:" hash on (case-sensitive) linux, and the "got:" hash on (case-insensitive) macOS. Was the "specified:" hash computed on a case-sensitive filesystem?
Well, at first I thought it's related but it's actually not. I'm following 22.05-release-channel. The mentioned commit never made it to my channel.
Steps to reproduce the behavior:
I tried imperatively install nix-2.9.1 and then tested why darwin-rebuild switch
again, but I'm not sure if it really picked up the nix-version from nix-env. At least it failed exactly the same way.
sudo -i nix-channel --list
darwin https://github.com/LnL7/nix-darwin/archive/master.tar.gz
home-manager https://github.com/nix-community/home-manager/archive/release-22.05.tar.gz
nixpkgs https://nixos.org/channels/nixpkgs-22.05-darwin
Please run nix-shell -p nix-info --run "nix-info -m"
and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
- system: `"aarch64-darwin"`
- host os: `Darwin 21.5.0, macOS 12.4`
- multi-user?: `yes`
- sandbox: `no`
- version: `nix-env (Nix) 2.8.1`
- channels(lockejan): `""`
- channels(root): `"darwin, home-manager-22.05.tar.gz, nixpkgs"`
- nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`
I wonder if we could catch the bug with a sanity script. Someone with good knowledge of find?
One way would be to canonicalize the list of filenames and check for duplicates:
❯ curl -fsSL https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5 \
| tar tz | tr A-Z a-z | sort | uniq -d
nixos-nixpkgs-f2537a5/pkgs/tools/x11/
nixpkgs was fixed in https://github.com/NixOS/nixpkgs/commit/aae0476728516c6a2ce78638fbc13436c4f4fa19
nixpkgs was fixed in NixOS/nixpkgs@aae0476
Indeed, but because I used the neovim-nightly-overlay it remained broken until its flake.lock file got updated to a point of nixpkgs where the fix got merged. I don't really understand it yet, since I don't use flakes, but @thefloweringash brilliantly figured this out and pointed to https://github.com/nix-community/neovim-nightly-overlay/blob/e82a3b43fd30507b8de1e11384282f0479926656/flake.lock#L58-L65 as the trouble making section.
I only updated my channels today and the problem finally disappeared.
One way would be to canonicalize the list of filenames and check for duplicates:
❯ curl -fsSL https://api.github.com/repos/nixos/nixpkgs/tarball/f2537a505d45c31fe5d9c27ea9829b6f4c4e6ac5 \ | tar tz | tr A-Z a-z | sort | uniq -d nixos-nixpkgs-f2537a5/pkgs/tools/x11/
Do you want to open a PR to add that to tarball validation steps?
TLDR; keep all of your binary cache history or don't use
binary-cache-v3.sqlite
:Turns out what happened:
/nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
binary-cache-v3.sqlite
/nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
/nix/store/nmhvs4pihm56qnhpb6j09ajcwxjx9h07-digest-0.0.1.2
binary-cache-v3.sqlite
had the old hashI presume this is due to binary non-determinism?
The fix is:
rm /nix/var/nix/binary-cache-v3.sqlite*