search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
16.43k stars 12.93k forks source link

meta.mainProgram set to nonexisting executable for some packages #298509

Open stuebinm opened 3 months ago

stuebinm commented 3 months ago

This is something of a post-scriptum for https://github.com/NixOS/nixpkgs/pull/297084: some packages in nixpkgs have meta.mainProgram set to a program name, but no executable with that name actually exists according to the unstable channel's programs.sqlite.

Unlike the mass-adding of meta.mainProgram for single-binary packages, I can't think of any easy way to fix theses cases, so I thought I'd at least open an issue about it.

Currently, there are 114 of these:

package attribute mainProgram $out/bin according to programs.sqlite
bsd-fingerd finger in.fingerd
cbor-diag cbor-diag yaml2cbor.rb, pretty2diag.rb, pretty2cbor.rb, json2pretty.rb, json2cbor.rb, diag2pretty.rb, diag2cbor.rb, cborseq2yaml.rb, cborseq2neatjson.rb, cborseq2json.rb, cborseq2diag.rb, cbor2yaml.rb, cbor2pretty.rb, cbor2json.rb, cbor2diag.rb
ccacheWrapper ccache-links strip, strings, size, readelf, ranlib, objdump, objcopy, nm, ld.gold, ld.bfd, ld, gprof, gcc, g++, elfedit, dwp, cpp, cc, c++filt, c++, as, ar, addr2line
certi certi certi.py
clang-tools clang clangd, clang-tidy, clang-scan-deps, clang-repl, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-pseudo, clang-offload-packager, clang-offload-bundler, clang-move, clang-linker-wrapper, clang-include-fixer, clang-include-cleaner, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_12 clang clangd, clang-tidy, clang-scan-deps, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-offload-wrapper, clang-offload-bundler, clang-move, clang-include-fixer, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_13 clang clangd, clang-tidy, clang-scan-deps, clang-repl, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-offload-wrapper, clang-offload-bundler, clang-move, clang-include-fixer, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_14 clang clangd, clang-tidy, clang-scan-deps, clang-repl, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-offload-wrapper, clang-offload-bundler, clang-nvlink-wrapper, clang-move, clang-linker-wrapper, clang-include-fixer, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_15 clang clangd, clang-tidy, clang-scan-deps, clang-repl, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-pseudo, clang-offload-wrapper, clang-offload-packager, clang-offload-bundler, clang-nvlink-wrapper, clang-move, clang-linker-wrapper, clang-include-fixer, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_16 clang clangd, clang-tidy, clang-scan-deps, clang-repl, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-pseudo, clang-offload-packager, clang-offload-bundler, clang-move, clang-linker-wrapper, clang-include-fixer, clang-include-cleaner, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
clang-tools_9 clang clangd, clang-tidy, clang-scan-deps, clang-reorder-fields, clang-rename, clang-refactor, clang-query, clang-offload-bundler, clang-include-fixer, clang-import-test, clang-format, clang-extdef-mapping, clang-doc, clang-check, clang-change-namespace, clang-apply-replacements
cloudrecon cloudrecon CloudRecon
codeberg-pages codeberg-pages pages
connmanMinimal connmanctl connmand-wait-online, connmand
coq coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqworker.opt, coqwc, coqtop.opt, coqtop.byte, coqtop, coqtimelog2html, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
coq_8_14 coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqwc, coqtop.opt, coqtop.byte, coqtop, coqtacticworker.opt, coqqueryworker.opt, coqproofworker.opt, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
coq_8_15 coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqwc, coqtop.opt, coqtop.byte, coqtop, coqtacticworker.opt, coqqueryworker.opt, coqproofworker.opt, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
coq_8_16 coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqwc, coqtop.opt, coqtop.byte, coqtop, coqtacticworker.opt, coqqueryworker.opt, coqproofworker.opt, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
coq_8_17 coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqworker.opt, coqwc, coqtop.opt, coqtop.byte, coqtop, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
coq_8_19 coqide votour, ocamllibdep, csdpcert, coqworkmgr, coqworker.opt, coqwc, coqtop.opt, coqtop.byte, coqtop, coqtimelog2html, coqpp, coqnative, coqidetop.opt, coqidetop.byte, coqdoc, coqdep, coqchk, coqc.byte, coqc, coq_makefile, coq-tex
corundum corundum corundum-skel
ddnet-server DDNet DDNet-Server
dep-scan dep-scan scan, depscan
distccWrapper distcc-links strip, strings, size, readelf, ranlib, objdump, objcopy, nm, ld.gold, ld.bfd, ld, gprof, gcc, g++, elfedit, dwp, cc, c++filt, c++, as, ar, addr2line
easyrpg-player `` easyrpg-player
exiflooter exiflooter exifLooter
frequest frequest FRequest
gnat-bootstrap gnat-bootstrap strip, strings, size, readelf, ranlib, objdump, objcopy, nm, ld.gold, ld.bfd, ld, gprof, gnatprep, gnatname, gnatmake, gnatls, gnatlink, gnatkr, gnatclean, gnatchop, gnatbind, gnat, gcc, g++, elfedit, dwp, cpp, cc, c++filt, c++, as, ar, addr2line
gnat-bootstrap11 gnat-bootstrap strip, strings, size, readelf, ranlib, objdump, objcopy, nm, ld.gold, ld.bfd, ld, gprof, gnatprep, gnatname, gnatmake, gnatls, gnatlink, gnatkr, gnatclean, gnatchop, gnatbind, gnat, gcc, g++, elfedit, dwp, cpp, cc, c++filt, c++, as, ar, addr2line
go2tv-lite go2tv go2tv-lite
go365 Go365 go365
graalvmCEPackages.graalnodejs js npx, npm, node
graalvmCEPackages.graalpy js python3, python, graalpy-lt, graalpy
graalvmCEPackages.truffleruby js truffleruby-polyglot-get, truffleruby, ruby, ri, rdoc, rdbg, rbs, rake, racc, irb, gem, erb, bundler, bundle
headphones-toolbox headphones-toolbox ploopy-headphones-toolbox
ivpn-service ivpn ivpn-service
klibcShrunk klcc zcat, uname, umount, true, sync, sleep, sh, run-init, resume, reboot, readlink, poweroff, pivot_root, nuke, nfsmount, mv, mount, mknod, mkfifo, mkdir, minips, ls, losetup, ln, kinit, kill, ipconfig, insmod, halt, gzip, gunzip, fstype, false, dmesg, dd, cpio, chroot, cat
ktls-utils ktls-utils tlshd
lanzaboote-tool lzbt lzbt-unwrapped
libgpg-error gen-posix-lock-obj yat2m, gpgrt-config, gpg-error-config, gpg-error
libtransmission transmission-cli transmission-show, transmission-remote, transmission-edit, transmission-create
libtransmission_4 transmission-cli transmission-show, transmission-remote, transmission-edit, transmission-create
mkvtoolnix-cli mkvtoolnix mkvpropedit, mkvmerge, mkvinfo, mkvextract
mumble mumble-server mumble-overlay, mumble
mupdf-headless mupdf mutool, muraster
netbird-ui netbird netbird-ui
nethack-qt nethack nethack-qt
nethack-x11 nethack nethack-x11
nucleiparser nparser nparse
nushellPlugins.net nu-plugin-net nu_plugin_net
nvmetcfg nvmetcfg nvmet
open-scq30 open-scq30 openscq30_gui, openscq30_cli
paralus-cli paralus cli
passwdqc passwdqc pwqgen, pwqfilter, pwqcheck
perl536Packages.Swim swin swim
perl538Packages.Swim swin swim
precice binprecice precice-tools, precice-profiling
pypy3 pypy pypy3.9, pypy3
pypy310 pypy pypy3.10, pypy3
python311Packages.inscriptis inscript.py inscriptis-api, inscript
python311Packages.subunit subunit-diff tap2subunit, subunit2pyunit, subunit2junitxml, subunit2gtk, subunit2disk, subunit2csv, subunit-tags, subunit-stats, subunit-output, subunit-notify, subunit-ls, subunit-filter, subunit-2to1, subunit-1to2
python311Packages.uplc opshin uplc
python312Packages.inscriptis inscript.py inscriptis-api, inscript
python312Packages.subunit subunit-diff tap2subunit, subunit2pyunit, subunit2junitxml, subunit2gtk, subunit2disk, subunit2csv, subunit-tags, subunit-stats, subunit-output, subunit-notify, subunit-ls, subunit-filter, subunit-2to1, subunit-1to2
riemann-tools riemann-tools riemann-zookeeper, riemann-varnish, riemann-proc, riemann-portcheck, riemann-ntp, riemann-nginx-status, riemann-net, riemann-memcached, riemann-kvminstance, riemann-health, riemann-haproxy, riemann-freeswitch, riemann-fd, riemann-diskstats, riemann-dir-space, riemann-dir-files-count, riemann-consul, riemann-cloudant, riemann-bench, riemann-apache-status
rocmPackages.llvm.clang rocm-llvm-clang size, readelf, ranlib, objdump, objcopy, nm, ld.lld, ld, dwp, clang++, clang, cc, c++, as, ar
rocmPackages_5.llvm.clang rocm-llvm-clang size, readelf, ranlib, objdump, objcopy, nm, ld.lld, ld, dwp, clang++, clang, cc, c++, as, ar
rrsync rsync rrsync
rubyPackages.certified certified certified-update
rubyPackages.cocoapods cocoapods pod
rubyPackages.diff-lcs diff-lcs ldiff, htmldiff
rubyPackages.jmespath jmespath jmespath.rb
rubyPackages.paru paru pandoc2yaml.rb, do-pandoc.rb
rubyPackages.ruby-graphviz ruby-graphviz xml2gv, ruby2gv, git2gv, gem2gv, dot2ruby
rubyPackages.ruby2ruby ruby2ruby r2r_show
rubyPackages.ruby_parser ruby_parser ruby_parse_extract_error, ruby_parse
rubyPackages.syntax_tree syntax_tree yarv, stree
rubyPackages.tiny_tds tiny_tds tsql-ttds, defncopy-ttds
rubyPackages_3_2.certified certified certified-update
rubyPackages_3_2.cocoapods cocoapods pod
rubyPackages_3_2.diff-lcs diff-lcs ldiff, htmldiff
rubyPackages_3_2.jmespath jmespath jmespath.rb
rubyPackages_3_2.paru paru pandoc2yaml.rb, do-pandoc.rb
rubyPackages_3_2.ruby-graphviz ruby-graphviz xml2gv, ruby2gv, git2gv, gem2gv, dot2ruby
rubyPackages_3_2.ruby2ruby ruby2ruby r2r_show
rubyPackages_3_2.ruby_parser ruby_parser ruby_parse_extract_error, ruby_parse
rubyPackages_3_2.syntax_tree syntax_tree yarv, stree
rubyPackages_3_2.tiny_tds tiny_tds tsql-ttds, defncopy-ttds
rubyPackages_3_3.certified certified certified-update
rubyPackages_3_3.cocoapods cocoapods pod
rubyPackages_3_3.diff-lcs diff-lcs ldiff, htmldiff
rubyPackages_3_3.jmespath jmespath jmespath.rb
rubyPackages_3_3.paru paru pandoc2yaml.rb, do-pandoc.rb
rubyPackages_3_3.ruby-graphviz ruby-graphviz xml2gv, ruby2gv, git2gv, gem2gv, dot2ruby
rubyPackages_3_3.ruby2ruby ruby2ruby r2r_show
rubyPackages_3_3.ruby_parser ruby_parser ruby_parse_extract_error, ruby_parse
rubyPackages_3_3.syntax_tree syntax_tree yarv, stree
rubyPackages_3_3.tiny_tds tiny_tds tsql-ttds, defncopy-ttds
rusty-psn-gui rusty-psn rusty-psn-gui
scss-lint scss_lint scss-lint
sensu sensu sensu-server, sensu-install, sensu-client, sensu-api, mutator-influxdb-line-protocol.rb, metrics-influxdb.rb, metrics-http-json.rb, metrics-http-json-deep.rb, metrics-disk.rb, metrics-disk-usage.rb, metrics-disk-capacity.rb, metrics-curl.rb, handler-show-event-config.rb, handler-logevent.rb, check-systemd.rb, check-smart.rb, check-smart-tests.rb, check-smart-status.rb, check-log.rb, check-last-modified.rb, check-journal.rb, check-influxdb.rb, check-influxdb-query.rb, check-https-cert.rb, check-http.rb, check-http-json.rb, check-http-cors.rb, check-head-redirect.rb, check-fstab-mounts.rb, check-disk-usage.rb
simdutf simdutf sutf, fastbase64
snowcrash SNOWCRASH snowcrash
ssh-copy-id ssh ssh-copy-id
synergy `` syntool, synergys, synergyd, synergyc, synergy
tang tangd tang-show-keys
taskjuggler taskjuggler tj3webd, tj3ts_summary, tj3ts_sender, tj3ts_receiver, tj3ss_sender, tj3ss_receiver, tj3man, tj3d, tj3client, tj3
terraform-landscape terraform_landscape landscape
toluapp tolua++ toluapp
topfew tf topfew
unrar-free unrar unrar-free
vial vial Vial
virtualboxHeadless VirtualBox VBoxManage, VBoxHeadless, VBoxBalloonCtrl
xcode-install xcode-install xcversion

A list can be generated as follows:


let
  pkgs = ./path/to/nixpkgs/master {};
  lib = pkgs.lib;

  dbFile = builtins.storePath <nixpkgs/programs.sqlite>;
  dbJson = pkgs.runCommand "json" { } ''
    ${lib.getExe pkgs.sqlite} ${dbFile} '.mode json' "
      SELECT package, system, json_group_array(name) AS names
      FROM Programs
      WHERE system = 'x86_64-linux'
      GROUP BY package, system;
    " > $out
  '';

  programs = lib.importJSON dbJson;

  # the channel is older than nixpkgs master, so some attributes may no longer exist
  # (but we edit on master, hence need that for e.g. positions)
  stillExists = row: lib.hasAttrByPath (lib.splitString "." row.package) pkgs;
  programs2 = lib.filter stillExists programs;

  getPackage = e: lib.getAttrFromPath (lib.splitString "." e) pkgs;
  isWrong = row:
    let
      mainProgram = (getPackage row.package).meta.mainProgram or null;
      names = builtins.fromJSON row.names;
      wrong = (!(builtins.any (name: name == mainProgram) names)) && mainProgram != null;
    in { inherit wrong names mainProgram; inherit (row) package; };

  wrong = lib.filter (a: a.wrong) (map isWrong programs2);
in
pkgs.writeText "wrong-mainProgram"
  (builtins.toJSON wrong)

which needs to be built as nix-build -I nixpkgs=channel:nixos-unstable file.nix so that it can import the programs.sqlite file of the channel (as in the above PR, this script is based on the one in https://github.com/NixOS/nixpkgs/issues/219567#issuecomment-1459241963).

Quite a number of these seem to be minor variations on a different package (e.g. ssh-copy-id, virtualboxHeadless) which inherit their meta attributes, but change which binaries are available.

Two cases even set mainProgram to "", owing to use of lib.optionalString.

fgaz commented 2 months ago

There is also at least one package (drawpile) that has multiple executables but was affected by the treewide change anyway (fix in #306734)

AndersonTorres commented 1 month ago

bsd-finger: https://github.com/NixOS/nixpkgs/pull/304404

jordan-bravo commented 1 week ago

Not sure if this is the best place to report but this warning is occurring with the package caddy-latest:

trace: warning: getExe: Package caddy-latest does not have the meta.mainProgram attribute.
We'll assume that the main program has the same name for now, but this behavior is deprecated,
because it leads to surprising errors when the assumption does not hold. If the package has a
main program, please set `meta.mainProgram` in its definition to make this warning go away.
Otherwise, if the package does not have a main program, or if you don't control its definition,
use getExe' to specify the name to the program, such as lib.getExe' foo "bar".
drupol commented 1 week ago

I don't have caddy-latest package.

And caddy has everything it needs: https://github.com/NixOS/nixpkgs/blob/f5afb674ec28f3647113ed7918adf8618de381df/pkgs/by-name/ca/caddy/package.nix#L76

Result: your nixpkgs is out of date, I guess.

jordan-bravo commented 1 week ago

I don't have caddy-latest package.

And caddy has everything it needs:

https://github.com/NixOS/nixpkgs/blob/f5afb674ec28f3647113ed7918adf8618de381df/pkgs/by-name/ca/caddy/package.nix#L76

Result: your nixpkgs is out of date, I guess.

I'm using a flake with nixpkgs/nixos-unstable as an input.

Moments ago, I ran nix flake update then rebuilt. The warning is still showing. I'm not sure what to make of that.

Edit: I'm using caddy as a NixOS module, if that is useful info: services.caddy.enable = true;

drupol commented 1 week ago

The issue is coming from your side. In nixpkgs, it has a meta.mainProgram since quite a while. You won't find the issue in nixpkgs but on your infra. I also checked the NixOS module, nothing special in there.

emilylange commented 1 week ago

There is no such thing as "caddy-latest" in nixpkgs. There was never and likely never will be.

It must come from your configuration or some input of that.

Edit: It's your own package @jordan-bravo. You will have to set meta.mainPackage yourself if you want the warning to go away: https://github.com/jordan-bravo/.nix/blob/cf93ce7978f4a002caf1adb78228310043e18a1f/nixos/shared/caddy.nix#L6

jordan-bravo commented 1 week ago

I am using a caddy plugin. I have a nixos file I've created with the following content:

# sovserv/caddy.nix
{ pkgs, config, plugins, ... }:

with pkgs;

stdenv.mkDerivation rec {
  pname = "caddy";
  # https://github.com/NixOS/nixpkgs/issues/113520
  version = "latest";
  dontUnpack = true;

  nativeBuildInputs = [ git go xcaddy ];

  configurePhase = ''
    export GOCACHE=$TMPDIR/go-cache
    export GOPATH="$TMPDIR/go"
  '';

  buildPhase =
    let
      pluginArgs = lib.concatMapStringsSep " " (plugin: "--with ${plugin}") plugins;
    in
    ''
      runHook preBuild
      ${xcaddy}/bin/xcaddy build latest ${pluginArgs}
      runHook postBuild
    '';

  installPhase = ''
    runHook preInstall
    mkdir -p $out/bin
    mv caddy $out/bin
    runHook postInstall
  '';
}

Note: this warning was not occurring prior to updating my flake input of nixos-unstable today. The previous time that it was updated was approximately 3 weeks ago, to put a timeframe on it.

emilylange commented 1 week ago

Edited my initial comment (again) right before you posted.

The important bit is

You will have to set meta.mainPackage yourself if you want the warning to go away: jordan-bravo/.nix@cf93ce7/nixos/shared/caddy.nix#L6

drupol commented 1 week ago

So basically you're rewritting a caddy derivation without meta attributes... and you're complaining that it's throwing a missing meta.mainProgram warning?

jordan-bravo commented 1 week ago

Thank you for the reply.

To set meta.mainPackage myself, would it look like this?

stdenv.mkDerivation rec {
  pname = "caddy";
  # ... other stuff
  version = "latest";
  meta.mainProgram = "caddy";
}

I'm not sure if that's the right value.

drupol commented 1 week ago

Yes. How about you trying before commenting?