= Solr Ref Guide in Asciidoc :toc:
This project is a proof of concept for reimplementing the https://cwiki.apache.org/confluence/display/solr[Apache Solr Reference Guide] as an Asciidoc-driven HTML site.
== ** AS OF 10 MAY 2017 THIS PROJECT HAS BEEN MERGED WITH SOLR SEE https://git1-us-west.apache.org/repos/asf?p=lucene-solr.git FOR LATEST CODE
OLD README CONTENT FOLLOWS
Instead of the current approach that uses Confluence, HTML pages and PDFs could be generated using a publishing toolchain that is integrated with existing Lucene/Solr build processes.
Asciidoc has been chosen as the format for Ref Guide format. This is a plain-text light-weight markup language, similar to Markdown, but with more options for structured articles, manuals, books, etc.
See the wiki page in this repo https://github.com/ctargett/refguide-asciidoc-poc/wiki/Why[Why Asciidoc] for discussion of reasons to change from Confluence and why Asciidoc is a good choice.
== What's in this Repo? This repo is a proof-of-concept for converting the Ref Guide to Asciidoc format. It includes:
solr-ref-guide
directory
This showcases what the "final" ref-guide files would look like in our Git repo for maintaining & building the Ref Guide.
Only this directory, and the files in it, would be copied/migrated into the official lucene-solr Git repo.confluence-export
directory
This contains tools for converting the raw confluence XML+HTML exports into .adoc
files (with images) which will be written into solr-ref-guide/src
.
This directory is also where the raw exports should be placed by folks interested in updating the converted docs:
confluence-export/raw-export
- put an unzipped confluence HTML export here
confluence-export/raw-xml-export
- put an unzipped confluence XML export herebuild.xml
* this top level build.xml
has a single convert-raw-confluence-exports
which handles converting the raw Confluence exports into confluence-export/cleaned-export
and then doing the final conversion to `solr-ref-guide/src/.adoc` files== Site Generation Process
Asciidoctor is designed as a toolchain for publishing content. Confluence provides a similar toolchain, so when thinking about replacing it, we must think about each aspect of the process and how it can be replaced.
Confluence provides a full end-to-end solution, from editing content to publishing a PDF. It includes features such as page comments and search, which are vital to community involvement and successful navigation of a topic as broad as Solr.
This project proposes the following:
.adoc
) format.
** These files will contain metadata attributes indicating the "hierarchical" structure of the documents..adoc
file metadata to build up navigation data filessidebar.json
file produced by our custom Java tooling during site build..adoc
file built by our custom java tooling which includes all "real" .adoc
files in the correct order.More on these options below - for specifics, see solr-ref-guide/README.adoc
.
=== Asciidoctor
Asciidoctor is a toolchain written in Ruby which facilitates converting text files to other formats for publishing.
Asciidoctor is both the markup language AND the conversion tool. The https://github.com/asciidoctor[Asciidoctor GitHub repositories] include several projects for build integration, content preview, content styling, and other tools required for a robust publishing solution.
=== HTML
It's possible to use Asciidoctor to generate HTML pages. However, the HTML output is limited to what is defined in the source content itself. There is no ability to add site navigation or add customized JavaScript. These more robust features can be added with a static site generator like Jekyll which is what we will be using.
=== Jekyll
Jekyll is a static site generator, which uses templates to create complete websites from raw content files in Asciidoc or Markdown (and other simple markup formats).
Jekyll is the most well-known of currently available static site generators, and has the most robust development and user community, and is the most mature.
Jekyll is written in Ruby and depends on native libraries, so it won't be viable to use JRuby to invoke jekyll via java, so building the HTML ref guide site will require that Jekyll be installed locally beforehand.
We're also using the https://github.com/asciidoctor/jekyll-asciidoc[`jekyll-asciidoc] plugin, which would also need to be installed locally as a dependency. The
build.xml` file still needs to be updated to try and detect if these gems are installed as part of the build process, and give a clean error w/install instructiones if they are not.
The current POC uses a Jekyll https://github.com/tomjohnson1492/documentation-theme-jekyll[Documentation Theme] developed by Tom Johnson, so the POC has a UX that is very different from out-of-the-box Jekyll.
I have additionally modified Tom's theme quite a bit:
jekyll-asciidoc
plugin (which his theme does not use). I've removed most of these features from the theme and rely instead on the Asciidoctor functionality.The borrowed theme implements "search", but it is not a full-text search engine. It's a keyword index of titles and page description metadata.
TO DO
build.xml
to download the Ruby dependencies (Ruby, Jekyll, and jekyll-asciidoc).We'll use the asciidoctor-ant
plugin which includes https://github.com/asciidoctor/asciidoctorj[`asciidoctorj] which itself includes
asciidoctorj-pdf, an implementation of https://github.com/asciidoctor/asciidoctor-pdf[
asciidoctor-pdf`] an Asciidoctor project to generate PDFs.
This means that even without Jekyll (or Ruby) installed, anyone will be able to build the Ref Guide PDF using jars that can be obtained from maven central via ivy.
There are some interesting issues that still need to be overcome:
asciidoctor-pdf
to make page breaks in sections configurable. Without this, page breaks are happening after every h2 level, which is used extensively in the source files. Thus, there are a lot more page breaks than there should be.As for styling, a YAML-based theme file is required to define colors, fonts, sizes, etc. This implements many of the features of CSS.
Overall, this plugin is the easiest to use of the various options (another option requires converting the content to DocBook format first, then to PDF), but is still in an alpha stage (as of 10 Nov 2016), so many features are still pending for future releases.
TO DO
== Comments
Comments are one of the two main reasons why a static site generator is required to have a full-featured replacement for Confluence.
This POC uses the Apache Comment System. See this repo's wiki page https://github.com/ctargett/refguide-asciidoc-poc/wiki/Comments[Comments] for more details on this system.
Because static site generators are template driven, it's simple to add JavaScript snippets to the template for each page. Variables allow filling in a page ID (this POC uses a page shortname) as each page is generated, which pull in the comments from the comment system.
Migration of existing comments from Confluence was briefly considered and rejected as too complicated. Comments are only available from the XML export from Confluence, while we needed the HTML export for effective content conversion. Then there is a question of if they are worth migrating - my own view is that they are not.
For more on how these decisions were made, see this repo's wiki page, https://github.com/ctargett/refguide-asciidoc-poc/wiki/Comments[Comments].
TO DO
== Open Questions
=== Location of Source
Should the content source live in a separate tree?
Should the content source live in a new sub-directory of the Solr Git repo?
=== Organization of Files
How should we organize the Ref Guide pages in the directory tree?
Some examples of how others have done it are available in this repo's wiki page https://github.com/ctargett/refguide-asciidoc-poc/wiki/FileOrganization[File Organization].
=== Hosting Options
Without Confluence, we will need to determine how and where to host the rendered pages. Some initial ideas:
. Host in ASF CMS with website. . Host however the javadocs are hosted.
=== Search
How will we provide search?
Recommend probably indexing generated HTML pages. Could use bin/post
from Solr to recurse over the HTML files and index them. In this case, we will need to figure out where to host Solr.