search

NixOS / infra

NixOS configurations for nixos.org and its servers
MIT License
209 stars 92 forks source link

Some paths listed in store-paths.xz are missing on cache.nixos.org #309

Closed dramforever closed 3 months ago

dramforever commented 7 months ago

Since early November 2023, a tiny number of paths listed in the store-paths.xz files from channels seems to be missing from cache.nixos.org.

Using nixos-unstable version nixos-23.11pre550515.e4ad989506ec, of the 177616 paths listed, 31 seem to be missing from cache.nixos.org.

List of missing paths ``` /nix/store/0ij1v44rs64nrrh7fzk9i8d5alfd5sxa-texlive-2022-env-info /nix/store/0y5vk84xh061qxidzvd118nvizmfjyca-texlive-2022-env-info /nix/store/3jaqnfl8nm99zmwiqlwv9dyhc8ws5saz-texlive-2022-env-info /nix/store/3mz1iwli104arhl50w46jsrnzvwm1dnr-texlive-2022-env-info /nix/store/4gskgc0zsg12k2b9yjxn8vglfi7y137c-texlive-2022-env-man /nix/store/6fbgn94nl9saqa02ngvpa25grnv5l5jz-texlive-2022-env-info /nix/store/7xmilmc02a1z6m2mg1lds1mznjvyf920-texlive-2022-env-info /nix/store/88p2177dxwgarb0z5vwvq7nr16cxc5bx-texlive-2022-env-man /nix/store/8aynfvjxq0ba641i9c0a0j3dlwz79wsz-texlive-2022-env-man /nix/store/a7v3ndrjibkwh15y6rb7a3bm4y72rh0i-texlive-2022-env-man /nix/store/aabvb66ws031vkwlbc71d804p0n7whr7-texlive-2022-env-man /nix/store/arprm2cmffqamd2kv8550vfrxmws5xga-texlive-2022-env-info /nix/store/av194b9pmqiqfia47d2p90pjaabf8pix-texlive-2022-env-info /nix/store/dsd0kq976a4w91cfgln75m3afhrmmd9w-texlive-2022-env-man /nix/store/fdwl6asaax5m2nwqb63kjygmnngi8php-texlive-2022-env-man /nix/store/fq9w6kaspwwkbjqnkjql0wjmxv0kqlv7-texlive-2022-env-man /nix/store/gfaajcw4wh0s2vf3rbi2rkvybhs30862-texlive-2022-env-info /nix/store/hswhxpmqwmfm3fbxr09yxann6jcc43gb-texlive-2022-env-man /nix/store/in0cc1sig5sckpm9zqc30anfin44cni0-texlive-2022-env-man /nix/store/jg17nmg54qggr83aln3amwlvrvaiycja-texlive-2022-env-man /nix/store/kprji6v01nyl0lcvsb30dx5zbjhf1kdi-texlive-2022-env-man /nix/store/lhq7mpg99rm0pv67b32mppdc8xsgab2v-texlive-2022-env-man /nix/store/mwkk39hyk3jwas8xkcwyhhildwfg0xps-texlive-2022-env-man ```

Interestingly all the paths have a name of texlive-2022-env-man or texlive-2022-env-info. Searching the other way around there is only one path matching this pattern that's actually on cache.nixos.org:

/nix/store/chmsspl3mz3f1lrq1cnqwihnpi6pqgn8-texlive-2022-env-man

It has been the case that all the paths are on cache.nixos.org since over three and half years ago, and this assumption seems to have only been broken recently. We have been relying on this file to provide mirror sites , now at: https://mirrors.bfsu.edu.cn/nix-channels/ and https://mirrors.ustc.edu.cn/nix-channels/, serving mainly mainland China users. This was the annoucement post back in March of 2020: https://discourse.nixos.org/t/announcing-tuna-nix-mirror/6144

We've now implemented a temporary workaround to ignore paths with these specific names.

(cc @delroth, who suggested filing an issue here in case there's a bug somewhere that matters)

delroth commented 7 months ago

cc @xworld21 - I suspect https://github.com/NixOS/nixpkgs/commit/7620b617e59c9d78119fa67081ce72efd9bc1b40 might be involved (based on the timing and the impacted paths). Can you think of any reason why this would make the man/info outputs for texlive envs behave weirdly like this on Hydra?

dramforever commented 7 months ago

Oh, in case this helps, we were last able to clone nixos-unstable on approximately 5 Nov (85f1ba3e51676fa8cc604a3d863d729026a6b8eb), and nixpkgs-unstable on 7 Nov (a0b3b06b7a82c965ae0bb1d59f6e386fe755001d).

xworld21 commented 7 months ago

Yes https://github.com/NixOS/nixpkgs/commit/7620b617e59c9d78119fa67081ce72efd9bc1b40 could definitely cause this. The .man and .info attributes are not true outputs: the underlying derivation has only output "out", which has no documentation. I emulate the multi-output by adding passthru.man, passthru.info and manipulating outputs, meta.outputsToInstall outside of the derivation, so that it works like a multi-output derivation when using nix-env and the likes.

So it seems like store-paths.xz is evaluating the top level attribute set as given, thus including the modified outputsToInstall instruction, while hydra is building only the underlying derivation -- on the assumption that the other outputs would be built automatically.

delroth commented 7 months ago

I wonder if we can fix it with a recurseForDerivations in the texlive env passthru? Disclaimer: I have no clue about any of this, I'm just guessing.

That seems very hacky, but tbf so does the actual change that caused those paths to not be built anymore... (not that I have a better suggestion)

xworld21 commented 7 months ago

I believe recurseForDerivations would cause duplicated outputs in search, and the search is already busy https://search.nixos.org/packages?channel=unstable&from=0&size=50&sort=relevance&type=packages&query=texlive (Not sure how to test this hypothesis, though.)

This is a problem we also have for TeX Live packages: right now Hydra does not build the texdoc containers, because we do not include them by default (they are huge), and we don't know any trick to convince Hydra to build them without also surfacing duplicates stuff in search.

Unfortunately this is a side effect of multiple outputs meaning two different things: parts that are built by a single derivation (nix-build) vs parts that can/should installed together (nix-env, nix-shell). TeX Live is a case where the latter makes sense, but the former does not, because the documentation is built independently.

dramforever commented 7 months ago

Would adding a level of indirection be acceptable? A multi-output derivation where out is symlinked to / copied from the main derivation and info/man from the docs derivation. This is somewhat wasteful on hydra but avoids having to do a hack.

xworld21 commented 7 months ago

Would adding a level of indirection be acceptable? A multi-output derivation where out is symlinked to / copied from the main derivation and info/man from the docs derivation

That would roughly match how TeX Live worked until last month. It's a bit wasteful but not dramatically so. It would affect any texlive.withPackages usage, though, including outside of Nixpkgs.

The other experiment I can try is removing "man" and "info" from .outputs. If nix-env and nix profile respect meta.outputsToInstall even if they are not listed, that could fix the specific mirroring problem (but not the problem that man and info pages will never be cached).

OTOH I am surprised that one man output was cached. How did that happen? Something we can reproduce maybe?

xworld21 commented 7 months ago

PS: can somebody point me at the code generating store-paths.xz? I want to see how the evaluation works (does it read .outputs, .all... ?).

xworld21 commented 7 months ago

The passthru.outputs hack is worse than I thought... nix-env -itexlive.withPackages (...)fails to install if themanoutput has not been built already. I didn't notice, because when testing the PR, I first built all the outputs, then checkednix-env`.

In summary: nix-env requires meta.outputsToInstall to be a subset of the actual build outputs, and so I need to tweak withPackages to be properly multi-output. This will fix the missing cache too. At this point, I'll need to check the new weird syntax nix profile install nixpkgs#foo^bar because I bet it is also hard-coded to look for derivation outputs instead of attributes.

dramforever commented 4 months ago

This issue still doesn't seem to be fixed, and now is also affecting texlive 2023. For example nixos-unstable version nixos-24.05pre584422.a4d4fe8c5002 has /nix/store/9wh8b44wil2xs8igcxn7yd24r562j0p2-texlive-2023-env-info which is not present on cache.nixos.org.

Is https://github.com/NixOS/nixpkgs/pull/270232 supposed to have fixed this? I thought so but evidently not...

xworld21 commented 4 months ago

The fix is one or two lines: remove passthru.outputs and/or passthru.all from build-tex-env.nix. I could take it all out, but I'd like to have the info output listed in search if possible.

So, question: how is store-paths.xz generated? I tried building closureInfo { rootPaths = [ texliveFull ]; } but there is no texlive-2023-env-info there.

vcunat commented 4 months ago

store-paths generation:

xworld21 commented 3 months ago

Assuming I haven't screwed up again, the store-paths.xz issue should have disappeared on both stable (23.11) and unstable. Can somebody please confirm this, and close the issue hopefully? I still don't quite know how to test it myself.

dramforever commented 3 months ago

I think this is fixed, we haven't got this problem for a while. Thanks a lot :slightly_smiling_face: