Supporting pandoc 2.11

[ickc]

There's a change in pandoc AST since 2.10. c.f. jgm/pandoc#1024, jgm/pandoc#6277 and https://github.com/jgm/pandoc/releases/tag/2.10

The new Underline is probably easy. The change of AST regarding table is huge (jgm/pandoc#1024) including support for span.

Edit: Links on new table AST

• PR
  • Feature request started here: jgm/pandoc#1024, jgm/pandoc-types#65
  • this is where the actual change of the AST happened: https://github.com/jgm/pandoc-types/pull/66/files?file-filters%5B%5D=.hs#diff-01f4ffe52cf097ab9ff89eebce394c86
  • pandoc 2.10 release note: https://github.com/jgm/pandoc/releases/tag/2.10
  • pandoc 2.11 release note (see HTML writer): https://github.com/jgm/pandoc/releases/tag/2.11
  • pandoc-types 1.21 changelog: https://github.com/jgm/pandoc-types/blob/master/changelog
• def
  • https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
  • doc on lua filters has some info on this: https://pandoc.org/lua-filters.html
  • See comments at https://github.com/jgm/pandoc-types/pull/66#issuecomment-611053332 and after on normalizing tables
• example ASTs:
  • https://github.com/despresc/pandoc-types/blob/master/test/test-pandoc-types.hs
  • https://github.com/jgm/pandoc/tree/master/test/tables

[tarleb]

@tarleb, is all the env var def documented somewhere?

You are right, this is only ever mentioned in the changelog, but not in any other documentation. I'll write a PR to add this to the JSON filter docs. (Edit: PR)

There is one other environment variable: PANDOC_READER_OPTIONS. This should contain the same information as the Lua global of the same name.

There is an open pandoc issue about exposing more info: jgm/pandoc#5221

[ickc]

Thanks for the info, I just double checked and only PANDOC_READER_OPTIONS, PANDOC_VERSION are available to JSON filters, so that's a bit different comparing to that documented in the lua filter section. PANDOC_READER_OPTIONS is going to be useful for us.

(But the info from the other 4 env var available to lua filter is either already known in JSON filters, or irrelevant for our case. So nothing's lost here.)

Looking forward to PANDOC_WRITER_OPTIONS.

[ickc]

pandoc 2.10 uses pandoc-types 1.21

pandoc 2.11 uses pandoc-types 1.22

IIRC something from pandoc-discuss mentioned a slight API changes is made to fix some consistency issue in pandoc 2.11.

So in short probably we'd jump to supporting 2.11 instead.

[fralau]

@sergiocorreia @ickc @tarleb Thanks a lot for this work for the community, this issue is getting really interesting ! Loads of things happened in this thread. After 50+ messages, perhaps someone could make a little summary of where we are all going with this issue, where we have come from, and where we are ?

[sergiocorreia]

@fralau

• Goal is to add support 2.11 soon (within the next couple of days)
• Going forward, we want to stay as close as possible to pandoc-latest , but avoid spending resources in keeping compat with older versions of pandoc. For that, people can just install an earlier version of panflute (so we will add a table indicating what version of panflute should they install if they have pandoc x.y)

Beyond that,

1. We want to expose a bit more metadata (the env vars that pandoc exposes)
2. We will probably need helper functions to make creating and modifying tables a bit easier, as 2.11 added quite a bit of complexity (still worth it though!). Right now, if you want to do something as simple as adding an extra column to a table, you would have to do a lot of changes to a lot of objects (you would need to touch every TableRow in TableHead, TableFoot, and all TableBody objects, plus adjust the ColSpec list)

[fralau]

@sergiocorreia Thanks a lot: this sounds very good !

Indeed, we might have an interesting problem in our project to determine (on the fly?) which version of pandoc is being used, so as to determine which version of panflute to use. Or else, ban versions of pandoc < 2.11 🤔

[ickc]

@fralau, regarding the lookup table between panflute and pandoc versions, for now it is just https://github.com/sergiocorreia/panflute/tree/stable#supported-pandoc-versions in the stable branch. It is an information for the end users and has to be acted upon manually.

Long term I'd like to have proper version control using conda. c.f. https://github.com/conda-forge/panflute-feedstock currently maintained by @kiwi0fruit. The pandoc dependence should be added there. And if done correctly people can do conda install panflute to have the matching version of compatible pandoc, panflute. (And say conda install pantable would have matching version of compatible pandoc, panflute, pantable and so on.)

For pip users, another option would be for panflute to bundle pandoc inside its wheel. I know that pypandoc used to do it in the past but has too much trouble that we wrote a download_pandoc function to just inform the endusers to run that function. However, it seems that they are building it from the source. Bundling the pre-compiled binaries in the wheels probably is much easier.

We probably should open another issue on this matter.

[ickc]

Here are the files which we currently use internally to test table output. The planets table tests colspans and rowspans, nordics checks element attributes and table footer, and students contains multiple table bodies, each with its own subheader.

nordics.txt planets.txt students.txt

GitHub requires a .txt ending, but all files contain pandoc JSON.

@tarleb, do you have a version of these with pandoc api 1.22 instead? Thanks.

[gabyx]

So good when this is implemented! :-) 👍 @ickc Do you know how much time this needs?. Thanks a lot for your effort.

[ickc]

@tarleb, do you have a version of these with pandoc api 1.22 instead? Thanks.

Nevermind, I converted them from pandoc 2.10.1 to 2.11.1.1:

nordics.txt planets.txt students.txt

Seems like only the JSON format has changed.

[sergiocorreia]

@tarleb, do you have a version of these with pandoc api 1.22 instead? Thanks.

Nevermind, I converted them from pandoc 2.10.1 to 2.11.1.1:

Is there a markdown source? In that way, we can generate the json each time we test; else the build might/will break next time pandoc_api changes even slightly.

[tarleb]

There are only .native sources: https://github.com/jgm/pandoc/tree/master/test/tables. Those can be converted to json with pandoc planets.native -o planets.json.

[sergiocorreia]

Thanks! Just added those files to the test cases!

[sergiocorreia]

By the way, the develop branch now supports the latest version of pandoc, will be adding that to master and releasing to pypi soon.

[ickc]

the master branch now passes tests including the native files pointed by @tarleb. Feel free to test the master branch, e.g. by pip install git+https://github.com/sergiocorreia/panflute.git@master.

[sergiocorreia]

Closing it; code is now on pypi

Also, definitely let us know if you find any bugs or have any ideas for easier table constructors. For instance, most filter creators probably don't want to specify table foot, header of table body, colspec, etc., so a possibility is a SimpleTable() that then populates the full-on Table().

[gabyx]

@sergiocorreia , @ickc, @tarleb Huge thanks for updating! Really appreciated!