Support installing manpages (and potentially other files)

[joshtriplett]

I use Cargo to build a command-line tool written in Rust. I've also written an accompanying manpage, command.1. I'd like to tell Cargo about manpages via Cargo.toml, so that cargo install will install them to the appropriate location (gzip-compressed and installed to ${root}/share/man/man1/command.1.gz, where the section 1 in man1 gets extracted from the first character of the manpage extension).

Note that once Cargo has a mechanism for a post-build script, some packages may wish to use that to build the manpage. For instance, a package might parse --help output to generate a manpage, process some other document language to produce a manpage, or use help2man-like functionality from an argument parser. However, for a first pass, I'd just like to have support for installing manpages already written directly in manpage format.

[Alhadis]

Just want to point out that man/*.[1-9] won't accommodate man pages that include an optional group-suffix (which are used prominently on SunOS-derived systems like illumos).

Though I get the impression you'd prefer to encourage simple, predictable extensions (and I can't blame you 😉).

FWIW, this is the regular expression used by GitHub to identify the most common man page extensions:

/\.(?:[1-9](?![0-9])[a-z_0-9]*|0p|n|man|mdoc)(?:\.in)?$/i

Users can set MANSECT to anything in their environments though, so… 🤷

[gasche]

A small ping on this issue with a concrete use-case: I am trying to use the superb-looking git-absorb program, which recommends cargo install git-absorb on systems (like mine) where it has not been packaged natively, but then I get an installation without a manpage and git absorb --help does not work. There is a manpage included in the repository, but it is not installed anywhere in ~/.cargo (related upstream issues: https://github.com/tummychow/git-absorb/issues/2 , https://github.com/tummychow/git-absorb/pull/25 ).

Concretely the lack of this feature means that git utilities written in Rust will always feel second-class, when installed through cargo.

My impression from reading the discussion so far:

• There is a clear consensus that cargo should not try to install manpages (or other resources) in their "native" location. Fine with me! I can set MANPATH to teach man to lookup in, say, ~/.cargo/share/man.
• There are discussions of how to automatically detect which files are manpages. Any choice that people agree on would be fine, one possibility being of simply listing files to be installed as manpages. (Especially if this directive can be used from build.rs or any other install script, then users with many manpages could perform globbing on their own for convenience. This sounds more robust than trying to agree on a specific regexp.)

[pmetzger]

The perfect is the enemy of the good. I'd suggest a nice simple piece of functionality (install man pages under a man/ directory under .cargo, as some have suggested) and try to produce "perfect" functionality someday in the future after there's some experience with the "imperfect" solution. Five years is already a long time to discuss an issue.

[devanoneth]

Anyway, doesn't cargo already add it's own directory ($HOME/.cargo/bin by default) to $PATH? AFAIK, man also checks for ../share/man relative to all directories in $PATH.

So, we could just make a man directory in $HOME/.cargo/share and start to make a man db there .e.g $HOME/.cargo/share/man/man1/somecoolrustclitool.1 for man somecoolrustclitool to work. I've tested this on macOS 12 and it works very well.

If the only remaining issue blocking this discussion is "where to place the man files", I don't think it should block it any longer. Using $HOME/.cargo/share/man would be quite idiomatic to how man works as a program.

[rrthomas]

AFAIK, man also checks for ../share/man relative to all directories in $PATH.

This depends on the man implementation; it is certainly true for man-db.

[BartMassey]

Every implementation of man I've ever seen supports $MANPATH. So there would be a reasonable fallback for implementations that didn't key off $PATH. We could probably get rustup to augment this along with $PATH in its setup.

[lilyball]

This depends on the man implementation; it is certainly true for man-db.

This is true for man-1.6g as well (which is what macOS uses). I'm not sure what other implementations are in common use, are there any that you know of that does not have this behavior?

Every implementation of man I've ever seen supports $MANPATH. So there would be a reasonable fallback for implementations that didn't key off $PATH. We could probably get rustup to augment this along with $PATH in its setup.

By "setup" do you mean rustup run, or do you mean when installing rustup? If the latter, please don't. I do not have $MANPATH set, I expect that most macOS users at least don't have it set either (I don't have sufficient Linux experience to know if this is common there).

Anyone that is using a man that does not infer already has to deal with this problem when augmenting their $PATH and so rustup is not special there.

If we're talking about the environment set up with rustup run then sure, set it there if you like, I don't imagine people are running rustup run stable man foo very often though.

[BartMassey]

It seems to me that given that installing rustup asks to modify PATH in your dotfiles, it may as well also ask to modify MANPATH if it is relevant? Users could of course opt out of either or both: I certainly don't let a rustup install edit my dotfiles. What am I missing here?

[lilyball]

I do not want rustup adding an unnecessary environment variable to my environment. Especially because $PATH inference is the generally correct behavior. If it sets $MANPATH and I subsequently modify my $PATH to remove the bin directory, I don't want man to continue showing me manpages for commands that are no longer in $PATH.

In the past I have seen tools decide whether to modify $MANPATH based on whether it is currently set. I don't know how applicable that is to the case of updating dotfiles, but it is a possibility, as the existence of (a non-empty) $MANPATH is a signal that setting it may be useful.

Asking users if they want to set it is a problem because many users are likely to not know much about $MANPATH (given that it's typically inferred) and may agree to set it when they don't need it.

I ask again, are there any known commonly-used man implementations that do not infer from $PATH? If this is purely a theoretical issue then it's not worth trying to solve. If there are known common implementations that require $MANPATH, perhaps we can test for that? Both macOS man and man-db have manpath commands IIRC (and I believe they both support man --path). rustup could check for this and use it to test if inference works, by creating the bin and share/man dirs and invoking this with $PATH set to include bin (and $MANPATH unset) and see if the new share/man dir shows up in the output. However I continue to believe this is unnecessary.

[rrthomas]

This depends on the man implementation; it is certainly true for man-db.

This is true for man-1.6g as well (which is what macOS uses). I'm not sure what other implementations are in common use, are there any that you know of that does not have this behavior?

My knowledge does not extend much beyond GNU, sorry.

[lilyball]

As a test, I fired up an Alpine Linux Docker image and ran apk add mandoc man-pages. The resulting man command does not appear to infer paths. man man gives me documentation that states that it is from the POSIX Programmer's Manual, and it includes the following paragraph:

   The historical MANPATH variable is not included in POSIX because no
   attempt is made to specify naming conventions for reference page files,
   nor even to mandate that they are files at all. On some implementations
   they could be a true database, a hypertext file, or even fixed strings
   within the man executable. The standard developers considered the
   portability of reference pages to be outside their scope of work.
   However, users should be aware that MANPATH is implemented on a number
   of historical systems and that it can be used to tailor the search
   pattern for reference pages from the various categories (utilities,
   functions, file formats, and so on) when the system administrator
   reveals the location and conventions for reference pages on the system.

This doesn't actually seem to be the correct manpage for mandoc though. I checked online and found https://mandoc.bsd.lv/man/man.1.html which does state that MANPATH is supported.

So basically, POSIX does not include MANPATH support in its specification, but it's supported by man-1.6g, man-db, and mandoc. Of those, man-1.6g and man-db infer search path from PATH whereas mandoc does not. All three of these implementations will print the current search path when invoked as man -w, which allows for testing whether path inference works if desired. I do not know offhand of any other man implementations to check.

All that said, I'm still of the opinion that anyone running a man implementation that does not infer paths is responsible for updating their own MANPATH (or /etc/man.conf) as appropriate.

[gasche]

For the record, I have a MANPATH on my Fedora machine that appears to have been refined by nvm and opam. Looking further, it looks like nvm (a Ruby package manager) adds to MANPATH when it is already defined (see the documentation) while opam (an OCaml package manager) adds to MANPATH on Linux, but not on OSX/BSD systems (see opamEnv.ml).

[gasche]

Personally I think that making any progress on this issue would be better than stalling because there is no perfect solution. If the idea to place manpages relative to bin is good enough for most systems, it sounds like a fairly good choice, especially given that:

• the location it suggests is the natural one in ~/.cargo
• users that have other man programs can always set MANPATH manually, which is very easy if there is a standard path for cargo installs.

[lilyball]

Nix also only sets MANPATH if it's already set in its profile script (and it also only does this for single-user installs; the multi-user install profile doesn't even bother to set MANPATH at all).

[BartMassey]

Nix also only sets MANPATH if it's already set in its profile script

I think this suggestion is reasonable: Rustup checks the environment for a MANPATH and offers to extend it if so. I'm also OK with staying out of the MANPATH business: like I say, I don't let Rustup touch my very ancient and convoluted dotfiles anyhow, so it won't affect me personally.

Anyhow, sounds to me like we can try to move forward now if somebody wants to do the manpage patches to Cargo. Messing with Rustup can wait until later, if ever.

[devanoneth]

Great to hear we have made some progress in understanding here. FWIW, brew (de facto macOS package manager) also extends MANPATH. We'd be aligning well with many other package managers.

[vilgotf]

Following XDG on linux and placing man pages under $XDG_DATA_HOME/man would be what I (as a linux user) would expect cargo to do, see #1734

[BartMassey]

@vilgotf Point taken. However, given that Cargo is already placing executables in $HOME/.cargo/bin, I think $HOME/.cargo/man makes the most sense after the discussion above. You'd be welcome to symlink $HOME/.cargo/man to $XDG_DATA_HOME/man if you like (on Rust 1.58.1 or later, of course :sweat_smile:).

[gasche]

I think we shouldn't tie the two discussions together ((1) installing manpages and (2) supporting the XDG specification). It's already hard enough to make progress on (1) (this issue will soon celebrate its 6th birthday), and it does not make sense to use XDG_DATA_HOME/man with all other files going to ~/.cargo.

I think the proper way to handle this is for interested people to push both discussions independently, without interfering with each other (after checking that doing one is not a fundamental blocker for the other), following the obvious "first merge wins" principle:

• if cargo changes to respect XDG by default first, we can change the man-install logic to respect it on XDG-using systems
• if cargo changes to install manpages by default, the in-progress XDG-support work can include a new change to the manpage logic (in addition to the other changes to cargo install directories).

[pmetzger]

I will agree with @gasche above and say again: the perfect is the enemy of the good. I suggest a very simple initial solution that gets slowly improved beats a never completed perfect solution. Even if the initial mechanism is extremely imperfect, it can be improved with experience.

[remyabel2]

Every project does things differently. For example, for exa I have to do:

pandoc --standalone -f markdown -t man $HOME/.cargo/registry/src/github.com-1ecc6299db9ec823/exa-0.10.1/man/exa_colors.5.md > ~/.local/share/man/man5/exa_colors.5

Whereas zoxide already has it in native format and I can just copy it over. And bat has a create.sh script that takes *.1.in files and creates *.1 files. Furthermore, it seems like some projects have man page generation disabled right now because of some issue with clap.

Typically the way that this would be handled on the distro package management side is that they generate the man pages via XYZ build steps and specify the manifest of files that are expected to be generated. For cargo the onus is obviously on the developer as they write the build scripts, I suspect that because everyone does things differently there will be situations where cargo will fail to properly install man pages. At this point though cargo is now just replicating the functionality that is already well handled by distro package managers, I suspect a more agnostic approach is to do what people have been doing for years with Makefiles and install assets to standard directories. Binaries go in $PREFIX/bin, assets/documentation/man pages go in $PREFIX/share and so on and so forth.

I am +1 on not overcomplicating things and just doing XDG_DATA_HOME/man. The sooner the better to avoid the need for hacky workarounds.

[matklad]

As a drive-by summary, it seems like there are some under-explored opportunities to improve the situation, which do not require coordination from Cargo, and which are not blocked on anything:

• https://github.com/rust-lang/cargo/issues/2729#issuecomment-279119341, document "suggested convention" for where in the repository man pages and such should be found. The value here is to cause various binaries to converge on the same convention, which removes mental overhead if you do packaging of many crates "by hand", and makes this potentially automatable. Example to learn from here is cargo-xtask, a polyfill for cargo workflows which, while a hack, did align the ecosystem somewhat.

  Action Item: consider all use-cases surfaced so far, write a spec which serves them, publish the spec as "advisory document", advertise the doc here, on reddit, and on irlo/urlo, and let the community iterate from a well-defined starting point.

• https://github.com/rust-lang/cargo/issues/2729#issuecomment-230861752 document "suggested metadata format in Cargo.toml", which opens further avenues for automation by, eg, distro packagers or a custom cargo command

• https://github.com/rust-lang/cargo/issues/2729#issuecomment-230861752 implement a custom cargo subcommand which take advantage of the above conventions to provide cargo install-like experience with manpages. With such a tool, a readme for hypothertical Rust cli utility called fabulous would look like this:

  # You can use `cargo install` to install the binary
  # Note: this won't install man pages or completions
  $ cargo install fabulous
  
  # To install with shell completions, use `cargo-install2`
  $ cargo install cargo-install2 && cargo install2 fabulous

  Existence of such cargo-install2 would make the experience of users of rust utilities better, and it would be a strong data point for eventual RFC and inclusion the functionality into cargo proper.

  Action Item: implement cargo install2, advertise that.

• As oftentimes manpages and completions are generated programmatically via a "post-build" script, documenting "suggested convention" for this is useful for all the same reasons. (to spitball, the convention might be "run cargo run --manifest-path ./build-dist/Cargo.toml to produce a ./dist folder, which would contain all the files to be packaged in a binary distribution")

cc @rotty, it seems that what you are doing with https://gitlab.com/rotty/cargo-parcel is basically this :-) The only thing missing is advertising the thing (eg, on this very issue), so that the community can run with the idea