Create man page

[micahellison]

Documentation Change

Experienced *nix users might expect to find detailed documentation by running man jrnl but it's not an option at this point.

Other Information

This was brought up a long time ago (#74) but the chief concern was not wanting to duplicate documentation, which is a good point.

The Fedora jrnl package does generate a manpage, apparently from the same documentation used for the site. This looks like a promising approach.

It's worth pointing out that Windows doesn't natively support man pages.

Ideally there would be a quick build test to make sure this is generating. I don't know what the manpage process looks like for package maintainers downstream, however.

[musicinmybrain]

Fedora package maintainer here. I’m using help2man, which generates output that is not perfect:

• It puts the usage under DESCRIPTION when it should be under SYNOPSIS, and calls the command __main__.py.
• The short description in NAME is not very useful
• The options are documented under DESCRIPTION instead of in an OPTIONS section.

Still, the formatting is decent overall, so I elected to stick with the generated output rather than writing a man page by hand and trying to track changes in jrnl’s --help output.

I know there are alternatives in Python-land, such as argparse-manpage or the man page support in Sphinx, but I haven’t tried any of them and have no opinion on their quality or usefulness.

---

If you decide you would rather have the best possible man page at the cost of duplicating documentation, I will be happy to contribute a hand-written one in groff_man(7) format (similar to the output of help2man).

---

From the perspective of a distribution maintainer, the best approach to installing the man page is just to make sure it is in the source tarball, and let us notice it and put it where it belongs. Man pages aren’t the kind of thing that Python packages can easily and reliably install. See https://github.com/pypa/packaging-problems/issues/72 for some relevant discussion.

If your man page is generated by a script or tool, I might or might not choose to re-generate it in the package build, but (in contrast to some other kinds of files, like minified JavaScript) I would not be required by distribution policy to re-generate it.

[wren]

I'm personally not against some amount of duplication in our docs. We already have some duplicated for the help screen (in jrnl/args.py), so it's not a big deal to me as long as there's an different intended audience (like the difference between someone browsing the website or looking at a man page).

My biggest concern here is maintainability. From what I see, the groff format is a bit terse (not too bad, but still a bit so). One of the ways I've seen less technical-leaning folks want to contribute to jrnl is by helping clean up the docs, so I'd like to keep that in mind as much ass possible.

So, my question is this: Is there a format/tool/whatever that leads to better man output, but is still readable and editable by someone less technical? Maybe there's something we could build into our CI pipeline to (hopefully) get the best of both worlds?

[musicinmybrain]

So, my question is this: Is there a format/tool/whatever that leads to better man output, but is still readable and editable by someone less technical? Maybe there's something we could build into our CI pipeline to (hopefully) get the best of both worlds?

There are a few alternatives out there that I know of, and probably a lot that I don’t know of:

• groff_man(7) format is indeed a bit terse, but at least the language is compact and well-documented, and there are a lot of examples. I can remember learning this format, and it wasn’t too bad, but I know a lot of people are put off by its appearance. I’m not sure how much of that is inherent to this format language, and how much is just an understandable lack of desire to learn Yet Another Documentation Format. It’s trivial to check your work with man ./mypage.1, which is nice.
• mdoc(7) is also used directly for man pages; it’s intended to express semantics rather than presentation, in contrast to groff_man. That’s an elegant idea, but I personally find it’s much less approachable than groff because it’s equally terse but is a larger language; its macros are much more numerous and more powerful. Here is a simple example of this format.
• AsciiDoc is fairly popular and has good man page support. AsciiDoctor seems to be the dominant implementation. I recently contributed a man page to another project and used AsciiDoc on the maintainers’ request. That was the first time I had worked with it. It looks more modern, or at least more approachable to those accustomed to Markdown, than groff_man, and it’s well-documented. I’m not sure if the learning curve is actually much easier than groff_man, but it’s not much harder. It’s certainly a good choice for projects or teams that are already using it for other documentation.
• I’ve seen xmltoman used, e.g. in pulseaudio. I haven’t worked with it, but it looks pretty straightforward, especially for an XML format.
• Sphinx has man page output support. To Python developers, this is attractive, since Sphinx is familiar. (To non-technical contributors, Markdown and reStructuredText are still Yet Another Documentation Format.) I am sure there are projects using Sphinx for man pages, but I have never noticed one, and I can’t find any documentation specifically talking about making nice or idiomatic man pages. That makes me wary.
• help2man works by running the program to be documented and formatting its --help output. Some GNU software uses it for the official man pages. It’s easy to get something that looks like a man page, but help2man generally only produces the highest-quality output if the --help output is carefully tuned to its heuristics. Here is an example of that. This eliminates duplicate documentation, but I personally don’t like the implicit constraints on the --help output, or the need to understand help2man before improving the --help output.

Not all options are equal, but I don’t think there’s one obvious best choice for all projects. Hopefully the above is a starting point for considering what’s best for jrnl.

[micahellison]

Thanks for all the info, @musicinmybrain. It seems like any doc converter that could use markdown as a data source would be ideal, since we use it everywhere else. Though I don't know what might be lost in the conversion process. Maybe pandoc could handle this?

I don't want Windows users to be left out by this also. The closest thing to man on native Windows is help, which outputs the following for help jrnl:

This command is not supported by the help utility.  Try "jrnl /?".

That makes me think that we could dump this manpage doc data to plain text and just include it as an undocumented output of jrnl /? on Windows.

[wren]

Ah, right. It's always good to have reminders that we support Windows, too.

So, from all of the above, here's what I think should happen. We should:

• Keep a hand-written man page in a more generic format (preferably markdown since that's what we use for everything else, but maybe asciidoc if non of the converters for markdown pan out).
• Take out what's currently on the help screen (in jrnl/args.py), and move it to this new markdown file. This will both reduce our bloated help screen, and give us a starting point for the new man page.
• Use a converter to generate:
  1. a man page for *nix users (in groff format)
  2. a plain text version for Windows (not in any specific format, but at least legible)
• Add whichever converter we choose to the CI, so that we don't manually have to keep running the converters
• Add a /? flag support to show the new page for Windows users. This should be hidden from the help screen for non-Windows users.

Does that sound right to everyone?

Also, outstanding questions in my mind:

1. Where do we store the markdown file? My gut says to put it somewhere docs, but maybe that's overloading that directory? On the other hand, that would make it easier to also host an online version of our man page.

2. Where do the generated files need to go? I usually see a man directory in the root with these files, but I'm not sure what's easiest for downstream package maintainers.

3. When do we run the converter? Are these files that should always be committed in the repo? Or are they only something that gets generated on release?

4. Are there any other formats we should convert to? We have the ones listed above, but this would be a good time to check to see if there are other formats that we'll need. We can also add more later as needed if nothing comes to mind now.

5. How to install man page for python packages? I really should know this, but I have no idea how to install man pages for users that install from pip or pipx. Also, probably homebrew? I'll try to look into this more, but if anyone can point me in the right direction, I would really appreciate it.

[wren]

Also, quick notes on some tools to investigate (in addition to everything mentioned above):

• https://rtomayko.github.io/ronn/ronn.1
• https://github.com/sunaku/md2man
• https://kramdown.gettalong.org/converter/man.html
• https://www.npmjs.com/package/markdown-to-txt

[wren]

@musicinmybrain What do you think? Are you still potentially up for helping write a man page?

[musicinmybrain]

Sure, I’m still willing to help. It does seem like you are still deciding exactly what you want to do, and how you want to do it.

[wren]

Okay, so I did research and testing (and found this guide very useful).

We should do the following tasks (and they can all be separate PRs by different people):

1. Write the source file for the man page
   • Written in markdown format
   • Stored at docs/man.md
2. Generate man pages from source file
   • Use pandoc to convert from markdown
   • Run in CI every time docs/man.md is edited, and commit into repo
   • Generate one set in roff format (for *nix)
   • Generate one set in plain text (for windows)
   • Store generated man pages in man directory (unless package maintainters like @musicinmybrain say a different place is better)
   • Ensure generated files are included on every release
3. Clean up help screen
   • Take out most things and make it a quick reference for commands and search filters
   • Put some notice in there about running man or /? to get the full docs

I still have no idea how to install man pages for python packages installed through pip/pipx, but we can figure that out later.

[viegasfh]

Hello, guys! I know at the moment we are using MkDocs for the documentation, but if we migrate to Sphinx, we can solve the problem for all platforms, as it allows producing the following output formats: HTML (including Windows HTML Help), LaTeX (for printable PDF versions), ePub, Texinfo, manual pages, plain text. MkDocs is limited to HTML, as it is a static site generator. I know Pandoc has been mentioned, but in my experience, Sphinx is easier to setup. The only disadvantage, if we can call it that, is that instead of Markdown it uses reStructuredText.

You can read more about it from the tool's page here

[viegasfh]

I forgot to add that reStructuredText is a much richer markup language that Markdown.

[micahellison]

I can see the appeal of switching to Sphinx, but I'm worried about migrating our docs pipeline from Markdown -- partly because of the labor to migrate, but also because I'm afraid it would break the version-specific documentation history in Read the Docs, which has been a little brittle for us already across the relatively long lifetime of this project. And since our site docs are in Markdown, I'd welcome the simplicity of keeping our man page in Markdown as well, as there is always a maintenance cost to adding languages to a project.

[viegasfh]

Hi. @micahellison ! Yes, it does make sense what you say, and I agree.

If the intention is to keep the existing Markdown workflow, perhaps the Pandoc suggestion is the more appropriate, but the question is how to generate the HtmlHelp file. I don't think Pandoc supports it. Unless there is an alternative that produces the htmlhelp file from the html files via the command line.

[micahellison]

the question is how to generate the HtmlHelp file

Are you referring to compiled HTML (CHM) Windows help files? I can imagine a process for generating this from HTML like you mention, though I think it would be fine enough to generate plaintext output to be retrieved by a /? argument. Basically, for rich text documentation, I think the website is sufficient, but I see the /? argument as the Windows analogue to the man page, as its purpose is to output more detailed help in the console.

[viegasfh]

Hum... OK. It makes sense.