FEAT: Adding a pdf pipeline to allow conversion of rst files to pdf

AakashGfude commented 5 years ago

This PR creates a new builder to read rst files and converts individual executed ipynb to pdf format using a latex_tempate, nbconvert and pandoc.

[x] pdf files for individual pages

book pdf file for the whole website (jupyter_pdf_book = True) [Future Feature] has been added as a tracked Issue #277

[x] The resulting pdf files should be located in _build/jupyterpdf/pdf/ (@AakashGfude)
[x] @AakashGfude provide summary of usage and options to help @mmcky with docs

Testing:

[x] check individual pdf files against collection made by ipynb_pdf_constructor (python) @mmcky
[x] add test cases to tests/pdf @AakashGfude

mmcky commented 5 years ago

[x] adjust travis to include xelatex (Issue #214)

mmcky commented 5 years ago

[x] @AakashGfude can we add compatibility with python27 by updating subprocess.run() to subprocess.call() if using python2?

Happy for you to check for python2 and raise NotImplementedError

mmcky commented 5 years ago

fixes #214

mmcky commented 5 years ago

@AakashGfude has found a resource to assist with adding a template approach to the pdf compilation chain. https://michaelgoerz.net/notes/custom-template-for-converting-jupyter-notebooks-to-latex.html.

This is a good way to go.

mmcky commented 5 years ago

@AakashGfude this doesn't need to be implemented for this PR but it is related (see Issue #229)

mmcky commented 4 years ago

@AnjuJoon @AakashGfude with the new additions to the Wiki page. Should we implement those in translate_all for the jupyter_target_pdf compilation option.

mmcky commented 4 years ago

[ ] Once happy with the solution being produce -- we need to update the reference ipynb for travis to pass for PR #211

mmcky commented 4 years ago

[x] add details of jupyterpdf builder to docs

mmcky commented 4 years ago

[x] add file filter option jupyter_pdf_excludepatterns for index* and 404 rst files etc. (@AakashGfude)
[x] add jupyter_pdf_urlpath option for internal links to other documents (@AakashGfude)

mmcky commented 4 years ago

@AakashGfude is it ok to execute pdf in parallel? Does each process maintain a different bib file for the references?

mmcky commented 4 years ago

[x] @AakashGfude are these warnings normal status codes from xelatex (i.e. should we be checking for a status code = 2)

[NbConvertApp] Writing 83075 bytes to /home/qebuild/repos/lecture-source-py/_build/pdf/jupyterpdf/executed/arma.tex
WARNING: xelatex exited with returncode 1 , encounterd in arma with error -- b''
WARNING: xelatex exited with returncode 1 , encounterd in arma with error -- b''
WARNING: xelatex exited with returncode 1 , encounterd in arma with error -- b''

this suggests warnings and errors are the same in xelatex exit codes. https://tex.stackexchange.com/questions/47912/xelatex-exit-codes. We may need to use this if we want to check for warnings vs errors: https://ctan.org/pkg/texloganalyser?lang=en

[x] can we save the log information from xelatex in the build location to refer to? (This is found in the executed directory

mmcky commented 4 years ago

[x] setup PR in lecture-source-jl for making-pdf branch.

Current bug for julia context reported: https://github.com/QuantEcon/lecture-source-jl/pull/701

mmcky commented 4 years ago

[x] copy _static to all subfolders during pdf compilation. (Removal is tracked in https://github.com/QuantEcon/sphinxcontrib-jupyter/issues/279 for 0.6.0)

mmcky commented 4 years ago

@AakashGfude I have opened a tracking issue for resolving the xelatex warnings

https://github.com/QuantEcon/sphinxcontrib-jupyter/issues/280

mmcky commented 4 years ago

~~thanks @AakashGfude the latest commit is working nicely. Just one item - the _static folder isn't required in the pdf folder if you wanted to not copy it across. Cheers. Matt~~ Actually I know why there is static there - as it is picking up the pdf files in the _static. Let's leave this for now and open a tracking issue to fix later.

mmcky commented 4 years ago

@AakashGfude working on test case for pdf. This warning is being flagged when not needed.

WARNING: Theme folder not present. Consider creating a theme folder for static assets

mmcky commented 4 years ago

@AakashGfude I think a lot of the current xelatex warnings are when there are no references to add to the References section which are appended to each file? We could check if any references are added and if they are then add the references to the latex file (or specify as an option for manual control)

mmcky commented 4 years ago

[ ] check reference solution for pdf test case with @AakashGfude

mmcky commented 4 years ago

[x] Does the extension check if xelatex is installed before executing? Perhaps we should add a check and issue a warning that pdf compilation is not possible without xelatex. I am happy to fix on using xelatex as that supports unicode. @AakashGfude I have added this in commit https://github.com/QuantEcon/sphinxcontrib-jupyter/pull/211/commits/194c78e637547411701e48c3477db3fe6898d81d at the builder stage. Is this the best place for this?

AakashGfude commented 4 years ago

[x] Does the extension check if xelatex is installed before executing? Perhaps we should add a check and issue a warning that pdf compilation is not possible without xelatex. I am happy to fix on using xelatex as that supports unicode. @AakashGfude I have added this in commit 194c78e at the builder stage. Is this the best place for this?

Looks good to me @mmcky . Thanks

AakashGfude commented 4 years ago

~thanks @AakashGfude the latest commit is working nicely. Just one item - the _static folder isn't required in the pdf folder if you wanted to not copy it across. Cheers. Matt~ Actually I know why there is static there - as it is picking up the pdf files in the _static. Let's leave this for now and open a tracking issue to fix later.

right. we can have an exclude patterns or we can store the subdirectories of lectures while we are building them, and only copy pdf's from those folders

mmcky commented 4 years ago

[x] @AakashGfude default behaviour when copying support files should be to overwrite. This bug crops up when running a second run using sphinx cache after modification of a single RST file.

Exception occurred:
  File "/home/qebuild/anaconda3/envs/latest/lib/python3.7/shutil.py", line 561, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/home/qebuild/repos/lecture-source-jl/_build/jupyterpdf/pdf/more_julia/_static/pdfs/pi2.pdf' already exists

mmcky commented 4 years ago

Looking into some bugs using julia - multi_agent_modles/markov_perf. @AakashGfude Do you know why

.. figure:: /_static/figures/judd_fig2.png
    :width: 70

Inventories trend to a common steady state

If we increase the depreciation rate to :math:`\delta = 0.05`, then we expect steady state inventories to fall

This is indeed the case, as the next figure shows

.. figure:: /_static/figures/judd_fig1.png
    :width: 70

is converted to in tex

\end{tcolorbox}

    \includegraphics{_static/figures/judd_fig2.png}\\
Inventories trend to a common steady state

If we increase the depreciation rate to \(\delta = 0.05\), then we
expect steady state inventories to fall

This is indeed the case, as the next figure shows

\begin{figure}
\centering
\includegraphics{_static/figures/judd_fig1.png}
\caption{\_static/figures/judd\_fig1.png}
\end{figure}

one figure is correctly converted to figure while the other is only an includegraphics with some form of line continuation \\

the intermediate markdown is:

![_static/figures/judd_fig2.png](_static/figures/judd_fig2.png)

Inventories trend to a common steady state

If we increase the depreciation rate to $\delta = 0.05$, then we expect steady state inventories to fall

This is indeed the case, as the next figure shows

![_static/figures/judd_fig1.png](_static/figures/judd_fig1.png)

which is confusing as they look the same. Is pandoc treating these differently for some reason?

[x] In addition I think we need to add [H] for figure placement in the preamble as in our case the figure shouldn't float but follow the discussion.

mmcky commented 4 years ago

[x] fix bug from datascience test set. We should make _static an optional copy.

Exception occurred:
  File "/home/qebuild/anaconda3/envs/latest/lib/python3.7/shutil.py", line 315, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/home/qebuild/repos/lecture-source-ds/_build/jupyterpdf/executed/_static'
The full traceback has been saved in /tmp/sphinx-err-qhby3ouu.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
Makefile:58: recipe for target 'pdf' failed
make: *** [pdf] Error 2

mmcky commented 4 years ago

[x] hey @AakashGfude can you please check https://github.com/QuantEcon/sphinxcontrib-jupyter/pull/211/commits/c87a4596cfc8b19c40d6ebb2b93d9b4279927a62 as I modified the condition to check for source folder and then copy to ensure latest version.

mmcky commented 4 years ago

[x] @AakashGfude why do we specify -D jupyter_latex_template="theme/templates/latex.tpl" in the Makefile. Shouldn't this be latex.tpl given we have template_path?
[x] Just trying to setup a branch on datascience for further pdf testing and get the following.

Exception occurred:
  File "/home/qebuild/repos/sphinxcontrib-jupyter/sphinxcontrib/jupyter/writers/process_latex.py", line 90, in main
    with open(filename,'r+', encoding="utf8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/qebuild/repos/lecture-source-ds/_build/jupyterpdf/executed/scientific/applications/classification.tex'

seems that the path is incorrect as this path doesn't exist: https://github.com/QuantEcon/lecture-source-ds/tree/master/src/scientific. This should just be applications?

AakashGfude commented 4 years ago

[x] @AakashGfude why do we specify -D jupyter_latex_template="theme/templates/latex.tpl" in the Makefile. Shouldn't this be latex.tpl given we have template_path?

@mmcky good catch. The branch started before that implementation, and went unnoticed. will modify it accordingly.

arnavs commented 4 years ago

I tried clicking an equation link in the McCall PDF and it pointed me to the website. Shouldn't it point me to the equation in the same PDF?

AakashGfude commented 4 years ago

[x] getting_started and julia_environment are in same folder getting_started_julia and the link in getting_started looks like this - interacting with julia, but it should have the subdirectory as well, for the website link to work

mmcky commented 4 years ago

@AakashGfude we still have some non general code in this PR we will need to address in writers/process_latex.py

    ## maketitle tex issue  need to change this 
    if(filename =="coase"):
         data = data.replace("Coase?s", "Coase's")

    if(filename == "dyn_stack"):
        data = data.replace("vs.~Stackelberg", "vs. Stackelberg")
        data = data.replace("vs.stackelberg", "vs-stackelberg")
    return data

and

data = data.replace("\\href{","\\href{https://lectures.quantecon.org/py/")

[ ] @AakashGfude to handle these cases in the translator

mmcky commented 4 years ago

[x] @mmcky initiate a final test run

QuantEcon / sphinxcontrib-jupyter

FEAT: Adding a pdf pipeline to allow conversion of rst files to pdf #211