Build PDF documentation on RTD

amotl commented 3 years ago

Introduction

In general, we would like RTD to provide Downloadable Documentation, i.e. a PDF file.

Issue

After bringing in #154, this apparently stopped working, so @gutzbenj reported at https://github.com/earthobservations/wetterdienst/pull/153#issuecomment-692570238:

It looks like Material for Sphinx does not offer to download the PDF version. Do you know if there's a way to get that working?

amotl commented 3 years ago

While the latest build apparently prepares everything appropriate for the LaTeX build step

The LaTeX files are in _build/latex.
Run 'make' in that directory to run these through (pdf)latex
(use `make latexpdf' here to do that automatically).

it croaks when doing so:

Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 256
Latexmk: Errors, in force_mode: so I tried finishing targets

The culprit might be this LaTeX error:

! LaTeX Error: Unknown graphics extension: .svg.

../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://github.com/earthobservations/wetterdienst/workflows/Tests/badge.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://codecov.io/gh/earthobservations/wetterdienst/branch/master/graph/badge.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://readthedocs.org/projects/wetterdienst/badge/?version=latest)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://img.shields.io/badge/code%20style-black-000000.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://img.shields.io/pypi/pyversions/wetterdienst.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://img.shields.io/pypi/v/wetterdienst.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://img.shields.io/pypi/status/wetterdienst.svg)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://pepy.tech/badge/wetterdienst/month)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://img.shields.io/github/license/earthobservations/wetterdienst)
../README.rst:: WARNING: a suitable image for latex builder not found: ['image/svg+xml'] (https://zenodo.org/badge/160953150.svg)

amotl commented 3 years ago

On https://github.com/sphinx-doc/sphinx/issues/4789, I found a hint about adding the Sphinx extension sphinx.ext.imgconverter.

After adding that to conf.py, Sphinx warned about

WARNING: convert command 'convert' cannot be run, check the image_converter setting

So, I've invoked brew install imagemagick and was able to make make latexpdf slightly more happy.

amotl commented 3 years ago

While on its way, make latexpdf halted processing and croaked with things like

(/usr/local/texlive/2020/texmf-dist/tex/latex/psnfss/t1pcr.fd)
Runaway argument?
{\sphinxincludegraphics {{code
! Paragraph ended before \sphinxhref was complete.
<to be read again>
                   \par
l.105

After finishing, it reported

Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  pdflatex: Command for 'pdflatex' gave return code 1
      Refer to 'wetterdienst.log' for details
----------------------
This message may duplicate earlier message.
Latexmk: Failure in processing file 'wetterdienst.tex':
   (Pdf)LaTeX didn't generate the expected log file 'wetterdienst.log'
----------------------
Latexmk: Use the -f option to force complete processing,
 unless error was exceeding maximum runs, or warnings treated as errors.

Invoking make latexpdf LATEXMKOPTS="-f" did not help.

amotl commented 3 years ago

There are some offending links the Sphinx LaTeX integration apparently didn't like.

$ grep -ri sphinxhref .

./docs/_build/latex/wetterdienst.tex:We originally started rebuilding \sphinxhref{https://github.com/brry/rdwd\textgreater{}}{rdwd} in Python as a starting project,

./docs/_build/latex/wetterdienst.tex:\sphinxhref{https://github.com/psf/black}{\sphinxincludegraphics{{code%20style-black-000000}.png}}

Things like

\sphinxhref{https://github.com/brry/rdwd\textgreater{}}{rdwd}
\sphinxhref{https://github.com/psf/black}{\sphinxincludegraphics{{code%20style-black-000000}.png}}

apparently make LaTeX very unhappy.

P.S.: It makes me sad that Black allows such a bad style for its very own badge URL ;]!

https://img.shields.io/badge/code%20style-black-000000.svg

amotl commented 3 years ago

I tried to fix all issues within #160. Enjoy

open docs/_build/latex/wetterdienst.pdf

However, we have to check whether the SVG conversion will also work on RTD. Is ImageMagick already installed there within the Python environment used for running Sphinx?

amotl commented 3 years ago

After fixing #161, we are now seeing at RTD Build #11889597:

Extension error:
convert exited with error:
[stderr]
b"convert-im6.q16: non-conforming drawing primitive definition `DejaVu' @ error/draw.c/DrawImage/3265.\n"
[stdout]
b''

It is probable that this is also related to the SVG conversion, see https://github.com/ImageMagick/ImageMagick/issues/974.

On my machine it still works. I have

$ convert -version

Version: ImageMagick 7.0.10-29 Q16 x86_64 2020-09-06 https://imagemagick.org
Copyright: © 1999-2020 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(3.1)
Delegates (built-in): bzlib freetype gslib heic jng jp2 jpeg lcms lqr ltdl lzma openexr png ps tiff webp xml zlib

On the other hand, RTD uses the standard imagemagick package from ubuntu:18.04, see Dockerfile.

$ convert -version

Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib

amotl commented 3 years ago

In this context, https://github.com/readthedocs/readthedocs-docker-images/issues/131 and https://github.com/readthedocs/readthedocs.org/issues/6919 as well as https://github.com/readthedocs/readthedocs-docker-images/pull/132 by @Daltz333 seem to be relevant.

He also created https://gist.github.com/Daltz333/3d0d4d04f45fd47604a98b60c2bf15b9, but apparently there is no official way to apply this as @humitos stated within https://github.com/readthedocs/readthedocs.org/issues/6919#issuecomment-617021093:

We don't officially support installing arbitrary packages at build time. Summarizing, there is no workaround for this at the moment more than continue talking about these packages in the other issue you have created.

We don't support installing packages using apt-get or other mechanisms. We should concentrate in the issue opened in the other repository.

amotl commented 3 years ago

Regarding issues with SVG conversions for PDF, there are already https://github.com/readthedocs/readthedocs.org/issues/905 by @PowerKiKi and https://github.com/readthedocs/readthedocs.org/issues/3979 by @ale5000-git.

Within these discussions, @mgeier recommended to use sphinxcontrib-svg2pdfconverter instead and @humitos implemented it bringing the librsvg2-bin and rsvg-convert programs into the mix through https://github.com/readthedocs/readthedocs-docker-images/issues/78 and https://github.com/readthedocs/readthedocs-docker-images/issues/79.

amotl commented 3 years ago

After enabling the recommended sphinxcontrib.rsvgconverter extension on my machine as outlined within 0be6516, Sphinx warned

WARNING: RSVG converter command 'rsvg-convert' cannot be run. Check the rsvg_converter_bin setting

which could easily be satisfied using brew install librsvg.

Then, the odd URL to the badge for "Black" hit again when invoking latexmk:

! Package pdftex.def Error: File `code%20style-black-000000.pdf' not found: using draft setting.

Now, even after removing this offending URL once more, it would still not work on my machine

Syntax Error: Couldn't read xref table
Syntax Warning: PDF file is damaged - attempting to reconstruct xref table...
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't read xref table

!pdfTeX error: pdflatex (file ./wetterdienst.pdf): xpdf: reading PDF image fail

Invoking brew install xpdf doesn't help either.

amotl commented 3 years ago

Apparently, the same thing also happens with the corresponding RTD Build #11890026.

Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't find trailer dictionary
Syntax Error: Couldn't read xref table

!pdfTeX error: pdflatex (file ./wetterdienst.pdf): xpdf: reading PDF image fail
ed
 ==> Fatal error occurred, no output PDF file produced!

Apparently, even when picking up and doing some conversion with the SVGs, something is not quite right for latexmk/pdflatex when trying to produce the PDF document.

It looks like we are stuck here. Bummer!

amotl commented 3 years ago

Actually, we don't want to embed these offending badge images into the PDF file at all. SO to the rescue, let's just include them for HTML by wrapping them into a .. only:: html directive, see Sphinx latexpdf ignore parts of a file.

The sad thing here is that this will make GitHub unhappy right away, see README.rst.

The nice thing is that the production of wetterdienst.pdf works well now.

amotl commented 3 years ago

I've worked around the issue by #162. However, the beautiful badges now are also gone for HTML Sphinx output. They are preserved on GitHub yet.

GitHub

RTD

gutzbenj commented 3 years ago

Thank you very much, Andreas! I think the way it works right now is just fine and we can close this! Badges did not work with the previous theme either and now even the citation is included in the documentation! Once again thanks!

rpatterson commented 2 months ago

FYI, I ran into this same issue with the Black badge. I went looking for an alternative badge but learned that the Shields.io static badge supports using underscores instead of %20. Changing the black badge's image URL to underscores seems to workaround the PDF build failure.

earthobservations / wetterdienst