TriBITSPub / TriBITS

TriBITS: Tribal Build, Integrate, and Test System,
http://tribits.org
Other
36 stars 46 forks source link

Address remaining blocking issues with Sphinx-generated TriBITS documentation and abandon DocUtils documentation #399

Open bartlettroscoe opened 3 years ago

bartlettroscoe commented 3 years ago

Blocked by: #386

Blocking: #400

Description

Sphinx-generated forms of the TriBITS documentation was put together in #386 and is used to auto-deploy Sphix-generated forms of the TriBITS reStructuredText documents on the GitHub Pages site https://tribitspub.github.io/TriBITS/ that are created and deployed by the GitHub Actions workflow deploy_docs.yml. However, there are a few issues the Sphinx-generated documentation that need to be worked out before abandoning the DocUtils-generated HTML files which generated using a cron job that runs every 20 minutes on the Linux server that serves up the site tribits.org. As a result, the end state of #386 was to keep both forms of the documentation generation in place until we can prioritize addressing the lingering issues with the Sphinx-generated documentation. Once we can address these last few issues, then we can drop the DocUtils generation of these documents and fully embrace the Sphinx generation of these documents and take full advantage of the features of Sphinx and really improve these online documents.

Definition of Done

List of known issues that need to be addressed

NOTE: Some of these issues could be fixed by the sphinx_rst_generator.py script before completing the transition and the implementation of the solution may not be all that different except for small details.

bartlettroscoe commented 3 years ago

One example of a formatting problem with the Sphinx-generated documentation are bulleted lists like:

https://github.com/TriBITSPub/TriBITS/blob/cab6deebe4e41896320f434108dc9c1aa312480a/tribits/core/package_arch/TribitsReadAllProjectDepsFilesCreateDepsGraph.cmake#L61-L69

With DocUtils, this creates the nice bulleted list HTML shown here:

which shows:

image

But the Sphinx-generated HTML for that same RST text shown here:

which shows:

image

Hum, that actually looks fine now. It did not look fine when I reviewed this 18 days ago in https://github.com/TriBITSPub/TriBITS/issues/386#issuecomment-876597266.

So I am not sure what happened there. There was an error in the generation of the HTML due to a defect that I fixed in commit 1d66677c09fc82db9eace7c7e9fb99832dad76b7. Perhaps that fixed some of the formatting problems I was seeing? Would that not be funny if that was true.

bartlettroscoe commented 3 years ago

There are a number of links between sections and anchors the different documents that currently assume they are deployed as HTML files side-by-side with the names TribitsBuildReference.html, TribitsUsersGuide.html and TribitsMaintainersGuide.html. For example, at:

https://tribitspub.github.io/TriBITS/users_guide/index.html#projectdir-ctestconfig-cmake

there is a link to "Dashboard Submissions" which currently points to TribitsBuildReference.html#dashboard-submissions which takes you to:

but when using the Sphinx-generated documentation, it needs to point to "../build_ref/index.html#dashboard-submissions" which takes you two:

An example going the other way is at:

where there is a link to "Package Dependencies and Enable/Disable Logic" which currently points to TribitsUsersGuide.html#package-dependencies-and-enable-disable-logic which takes you to:

but when using the Sphinx-generated documentation, it needs to point to "../users_guide/index.html#package-dependencies-and-enable-disable-logic" which takes you to:

After switching to Sphinx as the only documentation generation system, these links can be fixed statically with just a few search and replacements for *.rst and *.cmake files.

bartlettroscoe commented 3 years ago

So it turns out the problem that I am seeing with bulleted lists with the Sphinx-generated documentation seems to be a problem with the sphinx generation on my RHEL 7 Linux machine 'crf450'. On my RHEL 7 Linux machine with FireFox version 78.4.0esr (64-bit), the bulleted list at section:

shows:

image

The Sphinx-generated HTML on that machine is:

<p>On output, this creates all of the package lists and dependency
data-structures described in the section <a class="reference internal" href="#tribits-system-data-structures">TriBITS System Data Structures</a>
and more specifically the sections:</p>
<ul class="simple">
<li><p><a class="reference internal" href="#lists-of-external-and-internal-packages">Lists of external and internal packages</a></p></li>
<li><p><a class="reference internal" href="#list-variables-defining-the-package-dependencies-graph">List variables defining the package dependencies graph</a></p></li>
<li><p><a class="reference internal" href="#tribits-package-top-level-local-variables">TriBITS Package Top-Level Local Variables</a></p></li>
<li><p><a class="reference internal" href="#tribits-subpackage-top-level-local-variables">TriBITS Subpackage Top-Level Local Variables</a></p></li>
<li><p><a class="reference internal" href="#tribits-package-cache-variables">TriBITS Package Cache Variables</a></p></li>
</ul>

I copied the entire maintainers_guide/ directory generated on my RHEL 7 Linux machine with sphinx 4.1.0 back to my Windows 10 laptop and viewed it in Chrome 91.0.4472.164 and I also see the messed up formatting:

image

So it would appear there is something wrong with my sphinx installation on my RHEL 7 machine 'crf450'. Shoot. I am wondering if there is not something wrong with the "classes" used in the bullets simple and/or reference internal? I removed those from the HTML source to give:

<p>Macro run at the top project-level scope that reads the lists of packages
and TPLs and creates the packages dependency graph.</p>
<p>On output, this creates all of the package lists and dependency
data-structures described in the section <a class="reference internal" href="#tribits-system-data-structures">TriBITS System Data Structures</a>
and more specifically the sections:</p>
<ul>
<li><p><a href="#lists-of-external-and-internal-packages">Lists of external and internal packages</a></p></li>
<li><p><a href="#list-variables-defining-the-package-dependencies-graph">List variables defining the package dependencies graph</a></p></li>
<li><p><a href="#tribits-package-top-level-local-variables">TriBITS Package Top-Level Local Variables</a></p></li>
<li><p><a href="#tribits-subpackage-top-level-local-variables">TriBITS Subpackage Top-Level Local Variables</a></p></li>
<li><p><a href="#tribits-package-cache-variables">TriBITS Package Cache Variables</a></p></li>
</ul>

and it did not improve the formatting to bring back the bullets.

The full output from the command:

./tribits/doc/sphinx/build_sphinx_docs.sh &> build_sphinx_docs.out

on my RHEL 7 machine 'crf450' is shown below in build_sphinx_docs.out.

The only possible problem is the warning:

WARNING: html_static_path entry '_static' does not exist

but I don't know how that would impact the the generated HTML files. But if you look at the output from the GitHub Actions job here, you don't see that warning!

This is making me think there is something wrong with the Sphinx installation on my RHEL 7 machine 'crf450'. This makes me think it may not be so easy to develop and maintain Sphinx-based documentation if I can't build it on a RHEL 7 machine at SNL :-(

build_sphinx_docs.out (click to expand) . ``` Removing copied_files/ *** *** Generating Users and Maintainers guides ... *** Generating git version Generating list of Standard TriBITS TPLs ... Generating list of Common TriBITS TPLs ... Generating Directory structure of TribitsHelloWorld ... Generating output for 'checkin-test.py --help' ... Generating output for 'gitdist --help' and '--dist-help=' ... Copy updated file gitdist-dist-help-usage-tips.txt Generating output for 'clone_extra_repos.py --help' ... Generating output for 'snapshot-dir.py --help' ... Generating output for 'is_checkin_tested_commit.py --help' ... Generating output for 'get-tribits-packages-from-files-list.py --help' ... Generating output for 'get-tribits-packages-from-last-tests-failed.py --help' ... Generating output for 'filter-packages-list.py --help' ... Generating output for 'install_devtools.py --help' ... Copy TriBITS/README.DIRECTORY_CONTENTS.rst to TriBITS.README.DIRECTORY_CONTENTS.rst ... Generating HTML and PDF files for users guide ... /home/rabartl/Trilinos.base/Trilinos/TriBITS/tribits/doc/guides/users_guide Extracting TriBITS documentation from *.cmake files ... /home/rabartl/Trilinos.base/Trilinos/TriBITS/tribits/doc/guides Skipping final generation of 'users_guide' on request! Generating HTML file for maintainers_guide ... /home/rabartl/Trilinos.base/Trilinos/TriBITS/tribits/doc/guides/maintainers_guide Extracting TriBITS documentation from *.cmake files ... /home/rabartl/Trilinos.base/Trilinos/TriBITS/tribits/doc/guides Skipping final generation of 'maintainers_guide' on request! /home/rabartl/Trilinos.base/Trilinos/TriBITS/tribits/doc *** *** Generating TribitsBuildReference.[rst,html,pdf] ... *** Generating git version Only generating the file TribitsBuildReference.rst on request! make: `TribitsBuildReference.rst' is up to date. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build/html. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build/html. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build/html. DONE! ALL GOOD! ===> Generating Sphinx documentation: ===> Generating maintainers_guide ===> Generating users_guide ===> Generating build_ref ```
bartlettroscoe commented 3 years ago

So, I installed Sphinx and the Sphinx RTD theme on my Windows10 laptop using a Python and Pip installation designed for native windows with:

$ which pip3
/cygdrive/c/Python39/Scripts/pip3

$  pip3 install --local sphinx sphinx_rtd_theme

(note: sphinx_rtd_theme maps to sphinx-rtd-theme.)

and it installed:

$ ls /cygdrive/c/Users/rabartl/AppData/Roaming/Python/Python39/Scripts/sphinx-build.exe 
/cygdrive/c/Users/rabartl/AppData/Roaming/Python/Python39/Scripts/sphinx-build.exe

I was then able to build the TriBITS documentation with:

$ cd ~/TriBITS.base/TriBITS/

$ export PATH=/cygdrive/c/Users/rabartl/AppData/Roaming/Python/Python39/Scripts:$PATH

$ time ./tribits/doc/sphinx/build_sphinx_docs.sh &> build_sphinx_docs.out

real    0m28.610s
user    0m2.473s
sys     0m8.384s

(See detailed output in build_sphinx_docs.out below. )

This seemed to produce the documents:

$ find tribits/doc/sphinx/combined_docs/ -name "index.html"
tribits/doc/sphinx/combined_docs/build_ref/index.html
tribits/doc/sphinx/combined_docs/index.html
tribits/doc/sphinx/combined_docs/maintainers_guide/index.html
tribits/doc/sphinx/combined_docs/users_guide/index.html

And looking at that same section in the browser:

I see the correctly formatted list:

image

Note that the output in build_sphinx_docs.out still shows the warning WARNING: html_static_path entry '_static' does not exist but the HTML shows the correctly rendered bulleted lists. So that warning does not mean

So, it would seem that I can correctly build the Sphinx documentation on my Windows 10 laptop under cygwin but I can't on my super-fast RHEL 7. This is unfortunate but is still a workable solution at least for me (@bartlettroscoe).

build_sphinx_docs.out: (click to expand) . ``` Removing build_ref/_build/ Removing build_ref/index.rst Removing combined_docs/build_ref/ Removing combined_docs/maintainers_guide/ Removing combined_docs/users_guide/ Removing copied_files/ Removing maintainers_guide/_build/ Removing maintainers_guide/index.rst Removing users_guide/_build/ Removing users_guide/index.rst *** *** Generating Users and Maintainers guides ... *** Generating git version Generating list of Standard TriBITS TPLs ... Generating list of Common TriBITS TPLs ... Generating Directory structure of TribitsHelloWorld ... Generating output for 'checkin-test.py --help' ... Generating output for 'gitdist --help' and '--dist-help=' ... Generating output for 'clone_extra_repos.py --help' ... Generating output for 'snapshot-dir.py --help' ... Generating output for 'is_checkin_tested_commit.py --help' ... Generating output for 'get-tribits-packages-from-files-list.py --help' ... Generating output for 'get-tribits-packages-from-last-tests-failed.py --help' ... Generating output for 'filter-packages-list.py --help' ... Generating output for 'install_devtools.py --help' ... Copy TriBITS/README.DIRECTORY_CONTENTS.rst to TriBITS.README.DIRECTORY_CONTENTS.rst ... Generating HTML and PDF files for users guide ... /home/rabartl/TriBITS.base/TriBITS/tribits/doc/guides/users_guide Extracting TriBITS documentation from *.cmake files ... /home/rabartl/TriBITS.base/TriBITS/tribits/doc/guides Skipping final generation of 'users_guide' on request! Generating HTML file for maintainers_guide ... /home/rabartl/TriBITS.base/TriBITS/tribits/doc/guides/maintainers_guide Extracting TriBITS documentation from *.cmake files ... /home/rabartl/TriBITS.base/TriBITS/tribits/doc/guides Skipping final generation of 'maintainers_guide' on request! /home/rabartl/TriBITS.base/TriBITS/tribits/doc *** *** Generating TribitsBuildReference.[rst,html,pdf] ... *** Generating git version Only generating the file TribitsBuildReference.rst on request! make: 'TribitsBuildReference.rst' is up to date. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build\html. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build\html. Running Sphinx v4.1.0 making output directory... done WARNING: html_static_path entry '_static' does not exist building [mo]: targets for 0 po files that are out of date building [html]: targets for 1 source files that are out of date updating environment: [new config] 1 added, 0 changed, 0 removed reading sources... [100%] index looking for now-outdated files... none found pickling environment... done checking consistency... done preparing documents... done writing output... [100%] index generating indices... genindex done writing additional pages... search done copying static files... done copying extra files... done dumping search index in English (code: en)... done dumping object inventory... done build succeeded, 1 warning. The HTML pages are in _build\html. DONE! ALL GOOD! ===> Generating Sphinx documentation: ===> Generating maintainers_guide ===> Generating users_guide ===> Generating build_ref ```