generate HTML/markdown from vimdocs; make available on neovim.org

[justinmk]

User documentation should be made available on neovim.org as linkified HTML, like this but using the neovim.org theme, and the neovim-specific docs.

Only tool I could find that does this: http://vimdoc.sourceforge.net/cgi-bin/dumpvim2html.pl

Using the "html" target in the old Vim Makefile:

make html && ghp-import .

We may want to modify this script to output more disciplined HTML. Then that HTML could be fed to pandoc to open up pandora's box.

Next steps:

• [ ] See how pandoc likes the result of dumpvim2html.pl
• [ ] Optionally modify the output of dumpvim2html.pl to help pandoc
• [x] Apply neovim.org theme/CSS to generated vimdoc HTML
• [x] Make a build step for this, similar to doxgyen (#48)

[fwalch]

Did you get this script to work? What is the "tags" file? Also, make html (in vim, not neovim) doesn't work for me either:

$ make html
rm -rf dist/vim74html.zip
cd runtime/doc && zip -9 -z ../../dist/vim74html.zip *.html <../../dist/comment/74-html
    zip warning: name not matched: *.html

zip error: Nothing to do! (../../dist/vim74html.zip)
Makefile:599: recipe for target 'html' failed
make: *** [html] Error 12

[stefan991]

@fwalch I got it to work with these changes:

diff --git a/runtime/doc/Makefile b/runtime/doc/Makefile
index a6610a9..0e45952 100644
--- a/runtime/doc/Makefile
+++ b/runtime/doc/Makefile
@@ -11,7 +11,7 @@ VIMEXE = vim

 # include the config.mk from the source directory.  It's only needed to set
 # AWK, used for "make html".  Comment this out if the include gives problems.
-include ../../src/auto/config.mk
+# include ../../src/auto/config.mk

 DOCS = \
        arabic.txt \
@@ -34,7 +34,6 @@ DOCS = \
        gui_w16.txt \
        gui_w32.txt \
        gui_x11.txt \
-       hangulin.txt \
        hebrew.txt \
        help.txt \
        helphelp.txt \

The "tags" file is sort of a cross reference between the files, see :help tags. (the help command itself uses the tags file to find the locations in the documentation)

[stefan991]

Created a PR which includes this fix and some other cleanups: https://github.com/neovim/neovim/pull/1011

[ZyX-I]

Wondering why you say it is the only tool. There is also my format.vim for which you need colorscheme and non-trivial (but already written) setup and makehtml.awk from Vim main repository. I also heard of some python script for this job.

Perl script you are referencing here looks like a collection of hacks used for patching 2html output. My script is a heavily rewritten 2html which supports tags out of the box. Python script contains a parser written from scratch and must be the fastest solution.

Examples (every image links to the screenshotted page, screenshots from old Opera):

Awk script

Python script

format.vim

Cannot say for the other two, but in my format.vim all markup is completely redefinable, and there are even LaTeX and BBcode generators (though LaTeX one contains some issues (e.g. is not working fine with long lines) I am not fixing due to lack of interest). I do not like the idea of using a Perl script that does the job of processing tag file (already done by vim) and regex processing of HTML (you know one must not one parse HTML with regexes because it is not parsable with regexes). Python script or format.vim are probably the best choices here (awk output looks rather bad).

[ZyX-I]

One thing Python and other scripts do not provide, but my does LINK on mouse hover:

. I am clearly not a designer though.

[stefan991]

@ZyX-I Thanks for the summary of the tools. My next task on the todo list was to find something to replace the awk scripts that currently generate the docs. There is also a perl script in the doc folder: https://github.com/neovim/neovim/blob/master/runtime/doc/vim2html.pl, but I haven't looked at the output yet.

My favorite would be the python script as it is a single <300 lines python file, which should be easy to modify and maintain. Going though vim and format.vim and using a 140 lines shell script seems rather complex to me.

[ZyX-I]

@stefan991 This shell script is mostly doing auxilarly job: copying all documentation I want to format to some place, generating tag file for it (I do not want to depend on user (my) configuration and I do want to format only new help files), posting documentation, allowing to specify which one help file to format (processing arguments), working with job pool.

You absolutely need only less then fifteen lines for this job for specifying format.vim settings: Python script does not do posting (so these lines are out of comparison), job pool is provided by cmake backend (i.e. make or ninja), copying files to a separate location is not needed (you need to format only one documentation directory, not a collection of documentation directories from various plugins, and cmake backend also provides built-in way to limit script to only new files), arguments processing is not needed.

Do not say about 140 lines shell script: you will not have one.

[justinmk]

@ZyX-I Very appreciated. I should have remembered about http://vimhelp.appspot.com/ and looked there.

Whichever option can be set up on CI/travis most easily would be my choice, for now. Perhaps @c4rlo would be interested in helping us with his python script.

@fwalch has started styling the output here: https://github.com/neovim/bot-ci/pull/5

[c4rlo]

Hey guys, if you decide to go with my Python script then I'd be happy to help out. Hopefully it should be easy to tweak to your needs and set up. It uses an external CSS file for its styling, making it quite flexible.

[stefan991]

There is also my format.vim for which you need colorscheme and non-trivial (but already written) setup

.

This shell script is mostly doing auxilarly job:

@ZyX-I Seems I took non-trivial too serious.

[ZyX-I]

@stefan991 Actually I was pointing at specific line in the link. Unfortunately sourceforge stopped supporting pointing at specific lines since last time I checked. The setup is non-trivial because it is five (six with &runtimepath) Vim options and seven format.vim options: the problem is that a) defaults serve some general purpose of storing screen as-is (reducing performance a bit on startup: I should really have exchanged 2 of those options to -i NONE), but only formatting one file (meaning that tags that do not point to the current file are ignored and tags are referenced by line number and not name in anchors) in a meaningful location (by default full location is stored in <title>) and b) that tags all point to identifiers.

Last part is what was really non-trivial one: I needed to add precise regex that determines where each tag starts and ends as default (“any position that is not prefixed (start) or suffixed (end) by a keyword character”) was not serving well at all for help files because there are such interesting tags as . (literal dot). It is just 40 characters in two regexes, but figuring those forty characters was not very trivial task.

[ZyX-I]

So “non-trivial” did not mean “requiring lots of code”. It meant “requiring lots of research”.

[ZyX-I]

And research was actually lots of testing: deduce one regex, format HTML then check that resulting file contains all links I want to see there and contains no links I do not want to see. The last step (checking file) is the most time-consuming.