build pdf in quarto and bookdown - migrate to quarto

mitzimorris commented 10 months ago

Submission Checklist

[x] Builds locally
[x] New functions marked with `r since("VERSION")`
[x] Declare copyright holder and open-source license: see below

Summary

Use quarto to build individual pdfs of the 4 docsets - Stan User's Guide, Reference Manual, Functions Reference and the CmdStan User's Guide; also provide quarto-generated website for all manuals with top nav bar that lets users access all Stan Docs plus interface docs.

Changes:

latex tables instead of kablextra if table has mathjax
don't use macro '[` for equation arrays.
common header.tex and postamble.tex (for docs which have indexes) in src dir.
Table of Contents should have lowercase roman page numbers; main section has arabic numbers starting from 1
Ensure proper headers/footers everywhere. due to difference between quarto and bookdown, add latex directives '\mainmatter' to index.Rmd and index.qmd and then change pagestyle to plain for references.
light edits to overview.
ensure that subsections and subsubsections are either consistently numbered (functions-reference, cmdstan-guide) or unnumbered (stan-users-guide, reference-manual).
Functions Reference - remove hand-generated sets of links for chapter subsections, needed for bookdown html version, which renders each subsection as its own page.

Additions:

script to build pdf version of the docs using quarto

Copyright and Licensing

Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Columbia University

By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC BY-ND 4.0 (https://creativecommons.org/licenses/by-nd/4.0/)

mitzimorris commented 10 months ago

~~this PR is only for building individual pdfs with quarto instead of bookdown.~~

quarto makes a distinction between projects for websites and books. the per-document _quarto.yml files in the src directory subdirs are for projects of type book.

to build a quarto-based website which lets users switch between documents via a top navbar, will require a different _quarto.yml file, type website which will live in the _src directory.

mitzimorris commented 10 months ago

update:

to support both pdf versions plus a quarto-based unified docs website, added script build_quarto_pdfs.py which handles the quarto build logic.
added files and logic to src dir which will build a quarto website for all the docs.

this is what the landing page looks like:

docs_landing

this is what a page of the docs looks like: unified_docs

now that all docs are in same place, cross references seem to be broken everywhere. todo.

mitzimorris commented 10 months ago

update: this PR now builds both pdfs for individual docs and website for all docs. ready for review and/or beta testing.

latest change is to remove the html-version hack in the functions reference which provides a per-chapter psuedo-toc in the form of a set of links to the rendered html subsections. this isn't needed, given quarto right-hand-column toc. also upped the website toc depth to 3 - this only affects the right-hand-column toc - the sidebar nav shows chapters only.

~~still need to fix cross-refs - it's unclear whether book and websites both allow the kinds of hyperlinks described here: https://quarto.org/docs/books/book-crossrefs.html#hyperlinks~~

~~the SBC chapter could use some Goldilocks love w/r/t the figures - some render too big, some too small.~~

mitzimorris commented 10 months ago

remaining challenges:

how to update "link to latest" logic
test serving the html via github pages
update build scripts
make this repo a submodule of the stan-dev.github.io repo and serve docs and stan organization pages together in a unified website.

mitzimorris commented 9 months ago

~~big problem - quarto website vs individual pdfs per document directory - I'm not sure we can get links that work for both.~~

previously, both html and pdf were generated on a per-directory basis, and bookdown/knitr assembled all of the .Rmd files into a single .md/.tex file that was fed through pandoc. therefore, within-document links looked like internal anchors - e.g. - src/stan-users-guide/clustering.Rmd:

the [mixture modeling chapter](#mixture-modeling.chapter)).
...
As mentioned in the [clustering section](#clustering-mixture.section),

These links for for rendering the pdfs, because we're generating per-directory pdfs and for projects book, quarto assembles a single .tex file.

For the website, we need to change the links to include the filename, e.g.

the [mixture modeling chapter](finite-mixtures.Rmd)).

As mentioned in the [clustering section](finite-mixtures.Rmd#clustering-mixture.section),

After testing both pdf and website, quarto is doing the right thing. However, tracking down and updating the links in the Rmd files is going to be a total PITA.

WardBrian commented 9 months ago

What do you mean?

mitzimorris commented 9 months ago

What do you mean?

updated previous comment. bottom line: this is tedious, but doable. best done when brain dead.

mitzimorris commented 9 months ago

I don't know if I just now noticed this or if it wasn't like this before, but when I build the web version with the latest changes the top bar links to each doc set are drop downs, which don't really work that well with the number of sub-pages we have.

agree.

at some point I changed the config so that the nav bar entries mirrored the corresponding sidebar entries. bad idea.

mitzimorris commented 8 months ago

@WardBrian - ready for another review pass.

WardBrian commented 8 months ago

The latest commit has fixed the tab titles for me for the SUG, but the other 3 manuals still don't have the page title in the tab

mitzimorris commented 8 months ago

in latest commit fixed all page titles, added environment var file for STAN_DOCS_VERSION. still to do:

fix links
use includes for overview contents
navbar distinction between interfaces / other tools

mitzimorris commented 8 months ago

built the pdfs - the cross-document links are broken e.g:

Definite and indefinite one dimensional integrals can be performed in Stan
using the [`integrate_1d` function](../functions-reference/higher-order_functions.Rmd#functions-1d-integrator)

update:

web search turns up this discussion comment: https://github.com/quarto-dev/quarto-cli/discussions/3505#discussioncomment-6373686

tldr: links to other docs in the docset will be broken in the pdf.

trying one more thing: Stan Document set as monolithic pdf.

WardBrian commented 8 months ago

We shouldn't use relative links between pdfs (it doesn't currently for bookdown and I see no way this would ever work sensibly). In the current docs we just use full links to mc-stan.org/docs/.... when we need to link between two different manuals

mitzimorris commented 8 months ago

OK, full links between manuals everywhere. update: this seems to work, but having fixed the above set of links to match what they will be, they're broken now.

mitzimorris commented 8 months ago

ready for review. then we need to figure out how to launch the monolithic site.

mitzimorris commented 8 months ago

@WardBrian - this time for sure? double checked links, got rid of use of html anchor ids.

mitzimorris commented 8 months ago

should I add cleanup to this PR by getting rid bookdown config files _output.yml index.Rmd, book.json ?

WardBrian commented 8 months ago

You could, we will probably need some follow up PRs anyway at least for Jenkins, the readme, etc, so I think it’s also reasonable to wait

WardBrian commented 8 months ago

@mitzimorris the reference manual's quarto.yml got missed for the highlighting updates

WardBrian commented 8 months ago

@mitzimorris are you going to delete old index.Rmds here or would you rather leave that for another PR? Going to try to review again tomorrow morning

mitzimorris commented 8 months ago

I can do a cleanup commit.

mitzimorris commented 8 months ago

cleaned out all files not needed to use quarto to build the docs and modified the quarto build script to build the website. just realized the quarto build script args need to be reworked because the logic is off; the simplest fix would be either build the unified html website, or build one or more pdfs. ~~will try to fix tonight.~~

fixed. next up, deployment - cf https://quarto.org/docs/publishing/github-pages.html

mitzimorris commented 8 months ago

another thing that needs updating - links to latest.

WardBrian commented 8 months ago

I think it sufficies to decide where we want to put the link on the new pages and update the script

for the existing pages, it will be a similar process to what I just did in #709. For various reasons it may be easier to wait to do those until the new version is actually live online

WardBrian commented 8 months ago

I just pushed a change to link_to_latest.py, so I think that the only change needed for jenkins is to add Quarto to the docker image at https://github.com/stan-dev/ci-scripts/blob/master/docker/docs/Dockerfile

@serban-nicusor-toptal would you mind doing that? We can get a deb from https://quarto.org/docs/get-started/

So, after the above changes to build.py, I think we can:

Merge this
Re-build the 2.34 docs, based on Quarto.
Once those are built, we can do another pass using the automated link checker tools to update any places that need to point to the new urls Quarto will give them

WardBrian commented 8 months ago

Oh, and we need to rewrite the readme.

serban-nicusor-toptal commented 8 months ago

I just pushed a change to link_to_latest.py, so I think that the only change needed for jenkins is to add Quarto to the docker image at https://github.com/stan-dev/ci-scripts/blob/master/docker/docs/Dockerfile

@serban-nicusor-toptal would you mind doing that? We can get a deb from https://quarto.org/docs/get-started/

Tested and pushed to master, see https://github.com/stan-dev/ci-scripts/commit/36f3184662c41fb6a3a3524ecdde91b1693f4bfa

mitzimorris commented 8 months ago

users guide builds for me?

(base) ~/github/stan-dev/docs (issue/689-quarto-pdfs)> pdflatex --version
pdfTeX 3.141592653-2.6-1.40.25 (TeX Live 2023)
kpathsea version 6.3.5
...
Compiled with libpng 1.6.39; using libpng 1.6.39
Compiled with zlib 1.2.13; using zlib 1.2.13
Compiled with xpdf version 4.04
(base) ~/github/stan-dev/docs (issue/689-quarto-pdfs)> quarto --version
1.3.450

WardBrian commented 8 months ago

Have you already pulled? I fixed whatever issue I was having in 4c6d8c9

mitzimorris commented 8 months ago

pulled, merged, updated README

stan-dev / docs