Workflow for writing and exporting scientific documents with Pandoc

mitinarseny commented 5 years ago

This is not a bug or a feature request. It's just a discussion and sharing the experience.

Backstory

Last two months I spent on trying to develop a workflow for easy (but powerful) scientific writing in some lightweight markup language. Among other things that are implied by almost any markup language (code blocks, images, etc...) my needs were following:

Support for mathematical expressions
Good quality customizable PDF conversion without LaTeX (see more on that below)
HTML export (for deploying it to GitHub Pages) as PDF is printable paged media, but almost nobody these days cares about printing. PDF is good for offline reading, but why pages?

Firstly, I tried Jupyter: both Jupyter Notebooks and JupyterLab. After two weeks of trying to create a nice-looking PDF from notebook, I decided to stop this masachism and look for the alternatives. Here are some of the reasons, why Jupyter Notebooks is not appropriate solution for me:

Jupyter Notebook is buggy. JupyterLab is very buggy.
It is written in python, so automatically comes mistrust to this product.
Diffs are unreadable (I am aware of nbdime, but another tool for diffs, really?)

So, I discovered Pandoc. It is awesome and HTML support is excellent and highly customizable, but when it comes to PDF, it uses LaTeX. In my opinion, LaTeX engines is 40 years old ancient bulky piece of software and the only good part in it is its syntax for mathematical expressions. Of course, there are some "modern" LaTeX engines, but they suck, too. All LaTeX infrastructure seems weird for me and using it leads to committing a suicide (joke, do not worry :)).

Firstly, I wanted to read specification of PDF format and try to write custom pandoc Lua writer for pdf. Then I realized that pandoc supports only text writers, not binary, isn't it?

Then I discovered KaTeX and it's awesome.

Then I discovered CSS Paged Media Draft. It makes it possible to write PDF layout in CSS! And I thought it is a solution: it makes it possible to forget about LaTeX and write in indeed modern markup languages! But then I found out that there is only proprietary software that fully supports this standard and it costs a lot. (e.g. PrinceXML). Moreover, it does not support latest HTML and CSS features, so it sucks, too. There are two open-source solutions: wkhtmltopdf and weasyprint, but they are outdated in terms of latest standards, too.

I looked a bit more about different lightweight markup languages and found out about Asciidoctor. It is great for some reasons. For example, I was bribed by their cool VS Code Extension that being developed actively. But at the same time not as great as pandoc (pandoc flavored markdown gives you almost the same amount of control over the document as asciidoctor does). And there are some disadvantages:

AsciiDoc syntax is harder than Markdown
Their publishing stack seems a bit buggy. A good thing is that PDF files are made with native converter written in Ruby (bypassing LaTeX), but LaTeX math formulas are converted to SVG and pasted as images in PDF. It makes it impossible to search the document for text in formulas and sometimes SVGs are placed with wrong vertical alignment. Anyway, Ruby as any other interpreted and dynamically typed language isn't good choice for making stable and reliable software :)
Community is too small in comparison to pandoc's one. Anyway, despite having ~3k stars on GitHub, which is a lot, the only places that contain information on Asciidoctor is their website and GitHub Repo!

So, I switched back to pandoc and remembered why I went away from it. After two months of looking for the solution, I was so disappointed and desperate that I was going to switch to MS Word and do my homework there.

But then... I discovered Paged.js and now it seems like this is final solution! It brings CSS3 Paged Media into CSS2 and JS and makes it possible to use headless browser to produce PDF.

Final workflow

So, the final pipeline for me looks as the following:

+----------------------------+
|                            |
|  Pandoc Flavored Markdown  |
|                            |
+----------------------------+

              +
              |
              v           +-------------------+
                          |                   |
            pandoc  +---> |  HTML & CSS & JS  |  +--->  Chrome Headless + Paged.js
                          |                   |
                          +-------------------+                    +
                                                                   |
                                    +                              v
                                    |
                                    v                     +-----------------+
                                                          |                 |
                              GitHub Actions  <--------+  |       PDF       |
                                                          |                 |
                                    +                     +-----------------+
                                    |
                                    v

                            +----------------+
                            |                |
                            |  GitHub Pages  |
                            |                |
                            +----------------+

All these things are going to be tested in this repo.

Any ideas?

If there is alternative way of doing it simpler and more straight-forward, please, join the discussion.

denismaier commented 5 years ago

Why not use ConTeXt?

denismaier commented 5 years ago

Oh. And yes, Context is also tex-based. But I don't know what exactly you don't like about LaTeX so I can't tell if context is better or the same or even worse in your opinion.

mitinarseny commented 5 years ago

Why not use ConTeXt?

Thanks for the response! I didn't look towards this solution, as it contains word TeX :) Actually, the main point for exporting to PDF from HTML is that when you edit your code in markdown it's often desirable to see what happens to your final document. Compile PDF every second when you edit .md file is too heavy task for computer and it will slow down your writing process and it will lead to quick battery discharge if you using laptop. Moreover, most editor extensions displays your content in WebView, so It would be too inconvenient to have watch tasks for HTML and PDF exports simultaneously and two open windows with HTML and PDF preview as you want both to look perfect.

denismaier commented 5 years ago

Hmm, I don't really use previews and I tend not to compile until I am nearly finished. If i needed previews, I'd use HTML as a preview format and pdf only once your documents are in a good shape l.

mitinarseny commented 5 years ago

But I don't know what exactly you don't like about LaTeX

Here are some reasons:

Speed
Why not to load packages when you need them?
Why not to use package names that may contain package version and exactly where package can be downloaded from?
Why not to use requirements file and a convenient tool (tlmgr is inconvenient and buggy) that can read this file
Why not to use font instructions that says where to download fonts?
Why not to develop something like "fonts registry", where you can store both private and public fonts?
Why not to use modern documentation solution? Btw, where is the official documentation?)
Why not to use GitHub for the sake?

I agree, that, nowadays, LaTeX quality of generated PDF is the best of other present solutions, but I think it is time to use better modern and stable software that is being actively developed, rather than having infinite problems with package and font dependencies while dealing with ancient software like LaTeX.

denismaier commented 5 years ago

Ok. Regarding some of your objections against LaTeX ConTeXt might be a better solution. No packages, no tlmgr. These days you can every font installed on your system thanks to luatex. (This applies to LaTeX as well, of course.) Also it's monolithic. But your complaints about speed and Github will apply here as well. (But much LaTeX-code seems to be on Github. )

jgm commented 5 years ago

This isn't the right place for this sort of discussion. This tracker is for bug reports.

jgm commented 5 years ago

it is time to use better modern and stable software that is being actively developed, rather than having infinite problems with package and font dependencies while dealing with ancient software like LaTeX.

I think this statement is confused. LaTeX includes some "ancient" parts, like the original tex engine. This part is one of the most bug-free and well-tested programs out there. But LaTeX also includes some software that is being actively developed -- the packages you need to use to extend the default output. These parts tend to have bugs. So, the ancient software is relatively bug free and the new, actively developed software tends to have the bugs. Don't blame problems with LaTeX on its being "ancient"!

jgm / pandoc