The Interchange Development Group http://www.icdevgroup.org

ICDEVGROUP Documentation Set http://git.icdevgroup.org/?p=xmldocs.git;a=tree

This is a technical look at the XMLDOCS system, and it tells you how to get the documentation generated. For XMLDOCS authoring, see file WRITING.

INTRODUCTION

The (new) Interchange XML documentation set is completely self-contained. To build the complete documentation set, just run:

make git make

Sometimes you want to build multiple versions of documentation at once (for example, one online, one offline, one profiled for Debian GNU etc.). It is possible to change OUTPUT/ directory to something else (named OUTPUT) to achieve this effect. However, places where I couldn't have inserted variables have OUTPUT/ hardcoded, so OUTPUT is still made a symbolic link to the output dir of your choice.

Here's an example:

OUTPUT=-std make skel

After that point, you can omit OUTPUT= from subsequent calls to "make" and things will still clap together nicely.

Also, it is possible to generate documentation for a specific Interchange release, such as 5.0.0 or 5.2.0. To do so, use TARGET env variable:

TARGET="5.2.0" make refxmls

Just make sure to have TARGET defined on each 'make' invocation.

-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --

To build specific targets, see Makefile for target names. Few useful targets would include:

-- Those that are not part of 'make' routine: make git (to create complete sources/ directory, or update existing) make clean (removes OUTPUT/) make distclean (remove OUTPUT/ and tmp/) make look-clean (clean + 'mv tmp tmp.temporary'. Useful to only make the tree as if it is clean (to perform Git operations without errors), but then automatically rename the directory back to 'tmp/' and avoid the overhead of regenerating OlinkDB files).

-- And those that are part of 'make': make skel make cache make refxmls make olinkdbs-nc olinkdbs-c make OUTPUT/xmldocs.css make tmp/iccattut-nc.db tmp/iccattut-c.db make OUTPUT/iccattut.html OUTPUT/iccattut ...

See Makefile for a complete list, of course.

PREREQUISITES

To perform a successful build, the following programs and modules must be available:

• Perl
• Shell commands: mkdir, cp, ln, find, tar, gzip, bzip2, make
• git-core
• docbook-xml
• docbook-xsl
• xsltproc
• tetex-extra (and actually, packages it depends on)

Optional Perl modules:

• XML::Twig

(Install the Twig module if if you want to syntax-check your additions by calling i.e. "bin/refs-autogen --validate git-head").

RedHat setups

If running on Red Hat and not Debian GNU, apply this patch: http://icdevgroup.org/~docelic/xmldocs-rh.diff . It allows you to get usable results, although it's flaky and Debian GNU is really the preferred platform. (I suppose with a better patch, by someone know knows Red Hat and its DocBook XML setup, Red Hat would produce completely good results too - patches welcome).

On Gentoo GNU/Linux systems, the prerequisite ebuild packages are as follows:

• dev-lang/perl
• dev-util/git
• app-arch/gzip
• app-arch/bzip2
• app-text/docbook-xml-dtd
• app-text/docbook-xsl-stylesheets
• dev-libs/libxslt
• dev-util/ctags
• app-text/tetex
• dev-perl/XML-Twig

If running on Gentoo GNU/Linux then you will need to apply the following patch to correct the hard-coded, Debian-specific file locations:

http://icdevgroup.org/~kwalsh/xmldocs-gentoo.diff

FINAL OUTPUT

During the invocation of 'make', few files will be created:

docbook/auto*.ent - Files containing XML entities that we can use in our sources. For example, configuration directive Include is referenced simply as &conf-Include;, tag [include] is referenced simply as &tag-include;.

tmp/*.db - OLink DB files generated from source .xml files, and other, on-the-fly .xmls.

cache//* - Various Interchange source tree statistics, available over a filesystem interface. (For XInclusion in .xml sources and similar purposes). The files are generated by bin/mkreport which depends on cache//.cache.bin. cache//.cache.bin is generated by bin/stattree.

               The cache is Perl's portable Storable dump. It was
               originally kept in Git (so others could re-use it),
               but that didn't play out well in practice so now everyone
               building the docs needs to have the sources/ directory
               ready and run 'make caches' himself to get the .bin
               files generated.

OUTPUT/ - Autogenerated: directory containing the actual completely self-contained and interlinked documentation set. Once it's created, you can move it out of the build tree and package as you see fit.

               This can also be OUTPUT<yourprefix>, if you pass e.g.
               OUTPUT=-std in a call to make (as already shown above).

tmp/missing[2] - After you build the documentation, there will be a file named tmp/missing autogenerated, and it will contain a list of symbols with all the parts of the documentation they still miss for the item to be considered complete. tmp/missing2 will also be created that will list HOWTO items and glossary entries that need work.

DEVELOPMENT NOTES

The directory structure: Makefile - Main Makefile bin - Helper tools cache - Interchange source trees metadata docbook - DocBook XML support files files - Support files, such as downloadable examples etc. guides - Collection of guides refs - Collection of reference pages glossary - Collection of glossary items images - All images tmp - Scratch and temporary file space pending - A "pending" directory. If you have a chunk which you'd like to integrate in the docset, but don't have the time to prepare it yourself, just drop it in there and someone will pick it up. sources - (Not in Git.) run 'make git' to auto-create this directory with all needed contents. whatsnew - A directory containing whatsnew.xml, continuously updated what's new list. The items are added automatically; just place copies of the Git commit messages in the whatsnew/ directory; every time you run bin/whatsnew-update, it will process the files and update whatsnew.xml, which you can then push in Git.

Updating cache/: The dotfiles found in cache/ can only be generated when the sources/ directory is present as described above, and contains Interchange releases in directories named after release numbers (with the exception of "git-head"). Run 'make git' to create that sources/ directory with all the contents. Run 'make up-all' to invoke git update on all versions (or 'make up-' for specific). ('make git' can also be used to update repositories).

Leftover information, not important for standard procedure: **

• Once sources/ is in place, make sure all available versions are mentioned
• in Makefile's IC_VERSIONS variable, and then run 'make cache'.
• This will regenerate files for the versions you have.
• It is important to have as many releases as possible in sources/, because
• when you generate the documentation (reference pages most notably), the
• symbols there are considered "added" the first time they're encountered

• in the sources (so they'll appear more recent than they actually are). End of leftover information **

  When bin/mkreport runs later, it parses the .cache.bin file and produces number of output files (interesting "leaf nodes" in a hash). Those files are filesystem interface to tree-level statistics, and can be used in numerous ways, XInclude for example. Like: "Interchange consists of files". (Currently this is not available; bin/mkreport is outdated and broken, and will be fixed when I come to needing it).

  The XML "preprocessor" tool: There's bin/pp tool which you can use to write larger blocks of XML more conveniently. See the script itself for usage notes.

  The "CVS CO/UP" script: There's bin/coup tool that helps you manage sources/ directory. See the script or the Makefile for invocation examples. ('make git' and 'make up-<ver|all>' invoke bin/coup).

  Autogeneration of the reference pages: IMPORTANT Creation of new documentation parts: IMPORTANT

  When bin/stattree runs, it collects information about all the "symbols" in the source it can find (symbols are anything: pragmas, global variables, functions, tags, config directives ...). It collects the symbol names together with all files and line numbers (and few lines of context around them) where they appear. This is the first step of reference pages autogeneration.

  Some of the symbol information is derived from the source automatically; other parts must be added manually by us.

  Please note that the symbol name must match the symbol used in the source, although tags allowed to be used with two forms (dash and underscore).

  Let's take an example of "post_page" pragma (but the principle is the same for any symbol). User-supplied information is found in either:

  o refs// directory, or o refs/ file, where multiple sections are defined using the NAME

  and END tokens (similar to IC profiles ;-). Everything outside NAME ... END blocks is discarded and can effectively be used as a comment area.

  The refs/ single-file-based approach is now preferred. It's more convenient, and keeps the number of files in Git to a minimum. The bin/editem script advises to use it. Use refs//* only in special cases (which is never, or close to it).

  Regardless of the way you document an item, the following information is needed to consider the symbol documentation complete:

  • Pieces that must be user-supplied because defaults are only placeholders: purpose, synopsis, description, examples, see also

  • Pieces that could be user-supplied but have meaningful default: notes, bugs, authors, copyright

  • Autogenerated (can be overriden if really neccesary): id, name, symbol type, availability, source occurences

  Obscure information START /// Not needed in general

• All of above fields can both be overriden or appended with user-supplied
• information:

• *oo refs/ method (one-file, the preferred way):

• To fill the template of the reference page, you add content to sections
• in the following way:
• NAME section name
• section content
• END
• Over time it appeared we only want to append information and never
• override it, so this method does not have a way to override a value
• like refs//control (in multi-file method) can do.

• oo refs// method (one-directory, multiple-files):

• To unconditionally override values and/or provide one-liner contents, use
• refs//control file. It has pretty much inflexible
• "field: content" line format, but # comments can be used.
• To append with information, you use refs//, where is
• the name of an existing section, maybe followed by an arbitrary string.
• With the exception of example files, you generally only create one
• file for each section.
• To supply more examples, you could keep them in an informal structure
• like this:
• refs/post_page/example
• refs/post_page/example2
• refs/post_page/example-relative_pages
• refs/post_page/example:used-often
• refs/post_page/example.something
• (also, nothing prevents you from having more s in the same file
• if you like).
• You can't use # comments in the non-control files (they'd be left as-is),
• but you can use XML comments .
• To avoid having to escape all HTML entities and everything, simply
• enclose "dirty" blocks in <![CDATA[ ... ]]>. Obscure information END /// Not needed in general

To create the documentation for a yet non-existent item, your best bet is to start off by copying an existing item over.

When adding documentation entries, please favor QUALITY over QUANTITY. That means you should grep the whole Git repo for ALL information about a symbol (and supplement that with your own invaluable historical and technical information), and then write the item documentation that includes all that information. Also make sure to check the actual symbol source; at many places I've ** found undocumented options being present, and variables used or checked.

After you build the documentation, there will be a file named tmp/missing autogenerated, and it will contain a list of symbols with all the parts of the documentation they still miss for the item to be considered complete. tmp/missing2 will also be created that will list HOWTO items and glossary entries that need work. Clearing out this list is a priority; ** given that the new system is so convenient and cool, you have no excuse ;-)

Davor Ocelic, docelic@icdevgroup.org