passthru.tests is wrong after overrides

[roberth]

Describe the bug

passthru can not make use of the final pkg, leading to false test outcomes.

To Reproduce Steps to reproduce the behavior:

nix-repl> :b (cacert.overrideAttrs (_: { fixupPhase = "rm -rf $out/* $unbundled/*"; })).passthru.tests.blacklist-utf8
warning: unknown setting 'experimental-features'

this derivation produced the following outputs:
  out -> /nix/store/71z94ajbwvyq9jbf8avi38s8n3h4jq3w-verify-the-cacert-filter-output

Expected behavior

The above can be expected to fail, but the expression author did not have a means to use the overridden pkg, so we can't blame them.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context

Alternatives:

1. define packages as { stdenv }: let this = stdenv.mkDerivation { passthru.tests = f this; }; in this. This does not rely on the use of overlays, but doesn't support overriding.
2. define packages as { stdenv, mypkg }: stdenv.mkDerivation { passthru.tests = f mypkg; }. This relies on overlays to inject the right package. This makes the package correctness depend on overlays, which can not be assumed in general.
3. allow passing a function to mkDerivation, to make the final package available: { stdenv }: stdenv.mkDerivation (pkg: { passthru.tests = f pkg; })

Option 3 supports overriding and does not rely on its caller for passthru.tests to be correct.

Notify maintainers

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: [ "agda" "ammonite" "ammonite-repl" "ammonite_2_12" "ammonite_2_13" "appimage-run" "awscli" "babeld" "bat" "bazarr" "bazel" "bazel_0" "bazel_0_26" "bazel_0_29" "bazel_1" "bazel_3" "bazel_4" "binaryen" "bind" "bitwarden_rs" "bitwarden_rs-mysql" "bitwarden_rs-postgresql" "bitwarden_rs-sqlite" "bitwarden_rs-vault" "borgbackup" "buildbot" "buildbot-full" "buildbot-ui" "cacert" "caddy" "cage" "cagebreak" "calibre-web" "cassandra" "cassandra_2_1" "cassandra_2_2" "cassandra_3_0" "cassandra_3_11" "catatonit" "ceph" "ceph-dev" "charliecloud" "cjdns" "cni-plugins" "codimd" "colord" "conmon" "consul" "containerd" "convos" "corerad" "cri-o" "cri-o-unwrapped" "croc" "crun" "dconf2nix" "devicemapper" "dig" "discourse" "dnsutils" "docker" "docker-edge" "docker_20_10" "documize-community" "dovecot" "engelsystem" "envoy" "etcd" "etcdctl" "ferm" "firefox-esr-78-unwrapped" "firefox-unwrapped" "firejail" "fish" "flatpak" "flatpak-builder" "foot" "freeswitch" "ft2-clone" "fuse-overlayfs" "fwupd" "g4py" "gcab" "gdk-pixbuf" "gdk_pixbuf" "geant4" "gitea" "gitlab" "gitlab-ee" "gjs" "glib-networking" "glib-testing" "glib_networking" "gnome-photos" "go-neb" "google-compute-engine-oslogin" "gptfdisk" "grafana" "grafana-loki" "graphene" "graylog" "grocy" "grub2" "grub2_efi" "grub2_full" "grub2_light" "grub2_xen" "haproxy" "hedgedoc" "herbstluftwm" "hitch" "home-assistant" "host" "hub" "hydra-unstable" "ibus" "iftop" "inspircd" "inspircdMinimal" "installShellFiles" "iodine" "ipfs" "jasmin" "jellyfin" "jellyfin_10_5" "jenkins" "jicofo" "jitsi-meet" "jitsi-videobridge" "keepalived" "keepassx-community" "keepassx-reboot" "keepassx2-http" "keepassxc" "key" "keycloak" "lego" "libceph" "libgdata" "libjcat" "libtorch-bin" "libxmlb" "limesurvey" "link-grammar" "linux" "linux-libre" "linux-pam" "linux-rt" "linux-rt_5_10" "linux-rt_5_11" "linux-rt_5_4" "linux-rt_latest" "linux_4_14" "linux_4_19" "linux_4_4" "linux_4_9" "linux_5_10" "linux_5_11" "linux_5_4" "linux_hardened" "linux_hardkernel_4_14" "linux_hardkernel_latest" "linux_latest" "linux_latest-libre" "linux_latest_hardened" "linux_lqx" "linux_mptcp" "linux_mptcp_95" "linux_rpi0" "linux_rpi1" "linux_rpi2" "linux_rpi3" "linux_rpi4" "linux_testing" "linux_testing_bcachefs" "linux_xanmod" "linux_zen" "lobster" "logstash6-oss" "logstash7-oss" "lorri" "lsd" "lvm2" "lvm2_dmeventd" "malcontent" "mariadb" "mariadb-client" "matrix-appservice-irc" "matrix-synapse" "maturin" "memcached" "minecraft" "minecraft-server" "miniflux" "minio" "mkl" "molly-brown" "mpd" "mpd-small" "mumble" "mumble_git" "murmur" "murmur_git" "musescore" "mysql" "mysql-client" "nagios" "nano" "nanopb" "ncdns" "netdata" "nextcloud19" "nextcloud20" "nextcloud21" "nfs-utils" "nfsUtils" "nginx" "nginx-sso" "nginxMainline" "nginxShibboleth" "nginxStable" "nginxUnstable" "nologin" "nsd" "oh-my-zsh" "ombi" "openapi-generator-cli" "openapi-generator-cli-unstable" "openresty" "opensmtpd" "opentabletdriver" "os-prober" "ostree" "pam" "pdns-recursor" "phoronix-test-suite" "php" "php73" "php74" "php80" "pinnwand" "pipewire" "pleroma-otp" "podman" "podman-unwrapped" "pomerium" "postfix" "postgis" "postgresql" "postgresql95" "postgresql96" "postgresql_10" "postgresql_11" "postgresql_12" "postgresql_13" "postgresql_9_5" "postgresql_9_6" "powerdns" "privoxy" "prometheus" "prometheus-apcupsd-exporter" "prometheus-artifactory-exporter" "prometheus-bind-exporter" "prometheus-bird-exporter" "prometheus-blackbox-exporter" "prometheus-collectd-exporter" "prometheus-dnsmasq-exporter" "prometheus-domain-exporter" "prometheus-dovecot-exporter" "prometheus-fritzbox-exporter" "prometheus-jitsi-exporter" "prometheus-json-exporter" "prometheus-keylight-exporter" "prometheus-knot-exporter" "prometheus-lnd-exporter" "prometheus-mail-exporter" "prometheus-mikrotik-exporter" "prometheus-modemmanager-exporter" "prometheus-nextcloud-exporter" "prometheus-nginx-exporter" "prometheus-nginxlog-exporter" "prometheus-node-exporter" "prometheus-postfix-exporter" "prometheus-postgres-exporter" "prometheus-redis-exporter" "prometheus-rtl_433-exporter" "prometheus-smokeping-prober" "prometheus-snmp-exporter" "prometheus-surfboard-exporter" "prometheus-systemd-exporter" "prometheus-tor-exporter" "prometheus-varnish-exporter" "prometheus-wireguard-exporter" "prosody" "pt2-clone" "purescript" "pyo3-pack" "pypy" "pypy2" "pypy27" "pypy3" "pypy36" "python" "python2" "python27" "python27Full" "python2Full" "python3" "python310" "python36" "python36Full" "python37" "python37Full" "python38" "python38Full" "python39" "python39Full" "python3Full" "python3Minimal" "pythonFull" "qboot" "rabbitmq-server" "radarr" "redis" "restic" "robustirc-bridge" "rspamd" "rss2email" "rsync" "rsyslog" "rsyslog-light" "runc" "sage" "sageWithDoc" "sagemath" "samba" "samba4" "samba4Full" "sambaFull" "searx" "shadow" "shattered-pixel-dungeon" "signal-desktop" "slirp4netns" "slurm" "slurm-full" "slurm-llnl" "slurm-llnl-full" "smbclient" "snapcast" "snapper" "solc" "solc_0_8" "solr" "solr_8" "sonarr" "spago" "spike" "sqlite" "sqlite-interactive" "sqlite-replication" "sqliteInteractive" "sslh" "steck" "su" "sudo" "sympa" "syncthing" "syncthing-cli" "syncthing-discovery" "syncthing-relay" "teeworlds" "telegraf" "tor" "traefik" "tree-sitter" "trilium-server" "turbovnc" "ucg" "unit" "usbguard" "usbguard-nox" "uwsgi" "v2ray" "vault" "victoriametrics" "vscode" "vscodium" "wiki-js" "wireguard" "wireguard-tools" "wmderland" "wordpress" "wrapGAppsHook" "wrapGAppsNoGuiHook" "wsdd" "xandikos" "xdg-desktop-portal" "xmlsec" "xterm" "xulrunner" "yggdrasil" "yq" "yq-go" "zfs" "zfsStable" "zfsUnstable" "zigbee2mqtt" "zoneminder" "zsh-history" ]
  # all that have passthru.tests; lib.attrNames (lib.filterAttrs (k: v: let e = builtins.tryEval (v?passthru.tests && v.passthru.tests != {}); in e?value && e.value) (pkgs // { AAAAAASomeThingsFailToEvaluate = null; aleth = null; }))
# a list of nixos modules affected by the problem
module: [] # not directly affected

[stale[bot]]

I marked this as stale due to inactivity. → More info

[roberth]

Solution: #119942

[roberth]

Not merged. Please review #119942

[stale[bot]]

I marked this as stale due to inactivity. → More info