ickc commented 4 years ago

There's a change in pandoc AST since 2.10. c.f. jgm/pandoc#1024, jgm/pandoc#6277 and https://github.com/jgm/pandoc/releases/tag/2.10

The new Underline is probably easy. The change of AST regarding table is huge (jgm/pandoc#1024) including support for span.

Edit: Links on new table AST

PR
- Feature request started here: jgm/pandoc#1024, jgm/pandoc-types#65
- this is where the actual change of the AST happened: https://github.com/jgm/pandoc-types/pull/66/files?file-filters%5B%5D=.hs#diff-01f4ffe52cf097ab9ff89eebce394c86
- pandoc 2.10 release note: https://github.com/jgm/pandoc/releases/tag/2.10
- pandoc 2.11 release note (see HTML writer): https://github.com/jgm/pandoc/releases/tag/2.11
- pandoc-types 1.21 changelog: https://github.com/jgm/pandoc-types/blob/master/changelog
def
- https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html
- doc on lua filters has some info on this: https://pandoc.org/lua-filters.html
- See comments at https://github.com/jgm/pandoc-types/pull/66#issuecomment-611053332 and after on normalizing tables
example ASTs:
- https://github.com/despresc/pandoc-types/blob/master/test/test-pandoc-types.hs
- https://github.com/jgm/pandoc/tree/master/test/tables

sergiocorreia commented 4 years ago

Thanks! This is indeed an important change.

First of all, I feel like I should just add support for id/attrib/classes to all elements, because we seem to be converging towards that.

For the tables, is there an actual summary of the changes of the new AST? That issue thread looks huge

ickc commented 4 years ago

More links related to the AST:

https://github.com/jgm/pandoc/issues/1024
https://github.com/jgm/pandoc-types/issues/65
https://github.com/jgm/pandoc-types/pull/66 : this is where the actual change of the AST happened
https://hackage.haskell.org/package/pandoc

ickc commented 4 years ago

I feel like I should just add support for id/attrib/classes to all elements, because we seem to be converging towards that.

IIRC @jgm is reluctant to that idea. I don't think it will really happen, although many might ask for it. The solution seems to be just wrap whichever element one want to have attributes by a div and do it there.

jgm commented 4 years ago

If I were designing afresh, I'd have attributes on everything. But it would be quite a big change now.

sergiocorreia commented 4 years ago

@ickc by any chance do you have on hand any input file I can use to test the recent changes? I was initially thinking about having a markdown file with a complex table, but I understand that although the AST changed, the readers/writers haven't (yet).

So, are you aware of a way to receive and feed data into pandoc? One option is to try using a filter and seeing if pandoc accepts it, but seems a bit fragile.

ickc commented 4 years ago

I think may be the best way to obtain a test file would be starting from https://github.com/despresc/pandoc-types/blob/09cb4314010365abc4512c2363b83711c92ac18b/test/test-pandoc-types.hs. Here's the diff leading to this: https://github.com/jgm/pandoc-types/pull/66/files?file-filters%5B%5D=.hs#diff-01f4ffe52cf097ab9ff89eebce394c86. There's a JSON AST there that might be useful.

Other than the JSON, pandoc native AST could be a start. For short tables it is still manageable by hand.

ickc commented 4 years ago

Add some comments related to https://github.com/chdemko/pandoc-numbering/issues/18#issuecomment-655097180

According to https://github.com/jgm/pandoc-types/pull/66/files#diff-8c56b05df299ade82ea6265ee0b71bd0, these are added:

, Caption(..)
                               , ShortCaption
                               , RowHeadColumns(..)
                               , Alignment(..)
                               , ColWidth(..)
                               , ColSpec
                               , Row(..)
                               , TableHead(..)
                               , TableBody(..)
                               , TableFoot(..)
                               , Cell(..)
                               , RowSpan(..)
                               , ColSpan(..)

and TableCell is removed.

Then later,

         -- | A short caption, for use in, for instance, lists of figures.
 type ShortCaption = [Inline]

  -- | The caption of a table, with an optional short caption.
 data Caption = Caption (Maybe ShortCaption) [Block]

So the ShortCaption is an Inline and Caption is block element.

dilawar commented 4 years ago

Is there a beta version for this feature? I have pandoc-2.10 installed and if possible if would rather not downgrade.

NMarkgraf commented 4 years ago

As I can not see any commits toward a solution to this problem, is there any way we could help?

aubertc commented 4 years ago

The reasons why this is more challenging than thought have been sketched by @sergiocorreia , one of panflute's author, at https://github.com/chdemko/pandoc-numbering/issues/18#issuecomment-655097180 . If you find a good solution to the absence of test, I believe this could be extremely useful. You can also, probably, "hack" something, but there will be no guarantee that it will work on the long run (that is, when writers / readers will change too).

NMarkgraf commented 4 years ago

Okay, but so panflute is currently unusable ... I find that is also not a good idea...

ickc commented 4 years ago

To everyone who complain, Bare in mind you need to maintain your whole dependency stack. If you depends on panflute and panflute depends on pandoc<=2.9 for now, don't upgrade your pandoc and then complain. All of these open source developments are out of everyone's free time and is uncompensated.

This is really nothing out of ordinary whenever software libraries are used. Remember this whenever you want to spam any repos.

fralau commented 4 years ago

@sergiocorreia Clearly this is not your fault. The pandoc team thought they were doing something right by adding merged cells, but they did not anticipate the repercussions (IMHO this change was so significant it warranted a change of version of 3.0). Oh, well.

@ickc I did what you recommended (downgrade pandoc to 2.9). It was a bit of fuss with brew on MacOS, but that's doable (in case somebody wants indications, I kept a few notes). Plus I warned my colleagues about the issue.

I'd say we take this with philosophy. If there is something that the community can do to help, let us know.

ickc commented 3 years ago

The pandoc team thought they were doing something right by adding merged cells, but they did not anticipate the repercussions (IMHO this change was so significant it warranted a change of version of 3.0)

In fact it is a much anticipated feature for years. Pandoc's versioning model is major.major.minor.critical. So 2.9 to 2.10 is a major change that does not guarantee backward compatibility.

Many people surprised by something like this for the first time. But whichever software stack one depends on has to be managed can should not be blindly upgraded especially for major change that may break backward compatibility. Homebrew is one of the package manager that is not good at managing multiple versions (i.e. older versions.) For better control, you can use something like conda, conda create -n panflute pandoc<2.10 panflute -c conda-forge will gives you the necessary versions. OS level package managers often has this difference with those managing software stacks. In this case homebrew/macports/pacman from Arch, etc. can help you manage OS level software stacks, but other package manager such as conda can help managing your development software stacks.

While users of pandoc filters might not think that using them are "developing", but as long as they are using pandoc filters, they are developing in the sense that you have to manage the software stacks more carefully. That's partly why the idea of a pandoc package manager (pandocpm) died. In the case of this new table in AST, even the native filter framework embedded in pandoc failed. But in general the embedded framework should be the safest bet because it is officially supported and supposed to be upgraded in sync with pandoc itself.

ickc commented 3 years ago

By the way, the panflute package at conda-forge in principle can include dependence on pandoc: https://github.com/conda-forge/panflute-feedstock, maintained by @kiwi0fruit. With that it would be impossible to create a conflicted conda environment for panflute.

I know it is possible but don't know exactly how, @kiwi0fruit, any comments?

sergiocorreia commented 3 years ago

Hi Kolen, (and Laurent, Damien, etc.)

I agree with you that adding support for 2.10 (and now 2.10.1) is crucial. Due to family health issues, I am currently on leave from work and not in town, so even if this is important I have almost zero time to spend on updating.

One thing though: last time I looked my main problem was that I could not create or find an example of how the JSON is supposed to look, as no readers were currently supporting multirows and other features. Has this changed? If any of you can create a sample input that I can use that would be incredibly useful.

ickc commented 3 years ago

@sergiocorreia, I'm sorry to hear that and take your time to heal with your family! I totally understand and lately I don't have much time on these open source projects either. It's certainly not your responsibility to come up with a fix.

I know that there's PRs in pandoc on some readers/writers on the new table feature such as HTML, LaTeX, etc. And the test cases can provide some JSON to work with.

Anyone are welcome to step up to provide this or even make a PR too.

fralau commented 3 years ago

@sergiocorreia I would be willing to get samples, since that would also allow me to better understand what this evolution is about.

My only problem, right now, would be that I don't have access to a 2.10 platform on my Mac 🤷‍♂️ and I am not sure I wish to risk changing my setup (since brew is a pain for that, as far as pandoc is concerned). Anybody, any idea?

ickc commented 3 years ago

As I said you can use conda to manage an environment to control which pandoc version you’re using.

A simpler and more manual approach would be download the standalone binary release from GitHub (GitHub.com/Jgm/pandoc and then releases, among the 2 macOS releases, one is .zip, unzip it and you’ll find a pandoc binary inside. Call pandoc by specifying the path directly, say ./pandoc, then you’re using the pandoc downloaded, not the one in your PATH which is installed systemwide.)

tarleb commented 3 years ago

Here are the files which we currently use internally to test table output. The planets table tests colspans and rowspans, nordics checks element attributes and table footer, and students contains multiple table bodies, each with its own subheader.

nordics.txt planets.txt students.txt

GitHub requires a .txt ending, but all files contain pandoc JSON.

fralau commented 3 years ago

Right now I have no idea how spanned columns or rows are supposed to look like, so I haven't tested that.

I have generated a zip with a markdown source file (test.md) with an ordinary table, and produced outputs in the two formats: old pandoc and new(>=2.10):

native (hs)
json
html
markdown (md): to compare with the source.

pandoc_tables.zip

I have generated this with a little script (generate_output.sh), which I included so you could adapt it to your needs:

$ ./generate_output.sh
Old version: pandoc 2.9.2
New version: pandoc 2.10.1
Source file: test.md
------
Deleting pandoc_table*
Generate: json, old: pandoc_table_old.json...
Generate: json, new: pandoc_table_new.json...
Generate: html, old: pandoc_table_old.html...
Generate: html, new: pandoc_table_new.html...
Generate: native, old: pandoc_table_old.hs...
Generate: native, new: pandoc_table_new.hs...
Generate: markdown, old: pandoc_table_old.md...
Generate: markdown, new: pandoc_table_new.md...
Done!

Here is the start of the source:

Document with tables
====================

Table (normal)
--------------

+-------------+-------------+-------------+-------------+-------------+
| **Last      | **First     | **Address** | **City**    | -           |
| Name**      | Name**      |             |             |             |
|             |             |             |             | \*P         |
|             |             |             |             | ostCode\*\* |
+=============+=============+=============+=============+=============+
| Smith       | Minerva     | Ap          | Dangjin     | X48 8HQ     |
|             |             | \#421-2155  |             |             |
|             |             | Sed Ave     |             |             |
+-------------+-------------+-------------+-------------+-------------+
| Mcfarland   | Aaron       | 799-4032    | Cicagna     | N13 5XQ     |
|             |             | Cras Ave    |             |             |
+-------------+-------------+-------------+-------------+-------------+

I hope this can be useful

Note on MacOS

To use the 2.10 version, I followed @ickc's advice (thanks!) I downloaded the 2.10 zip file for MacOS. I unzipped it and found the executable in the bin directory.

I then referred to it, in the commands, with its path, instead of the pandoc command. Just to be sure I had the right version, I used pandoc's -v option.

tarleb commented 3 years ago

One more note: I've also started work on a new tool ast-migrator, which will help to deal with situations like this in the future. It can convert between pandoc AST version 1.20 and version 1.21, with more versions to follow. The idea is to plug the tool into the filter such that the input is converted to the old format, processed as before, and the result converted back into the new format before it is returned to pandoc. My Python is a bit rusty, so the following is mostly pseudo-code:

def main(doc=None)
    input = subprocess.check_output(['ast-migrate', '--down'], sys.stdin)
    output = io.StringIO()
    run_filter(actions, doc=doc, input_stream=io.StringIO(input), output_stream=output)
    subprocess.call(['ast-migrate', '--up'], output)

The tool has not been released yet. If somebody would like to test it, please ping me.

ickc commented 3 years ago

@tarleb, I think that's useful.

I think a helper like that is better to be inside panflute, and a keyword arg of the run_filter functions can specify the expected AST version of the filter (i.e. say a filter called pantable, it calls the function in panflute with that AST version specified and in panflute a helper is going to handle that automatically.) It reduces some boilerplate and make it easier for people to adopt.

There may also be an env var overriding this behavior, e.g. when pandoc is updated and panflute is yet to catch up then this env var can be used to override the assumed AST version.

May be open an issue on this after ast-migrator is released? (Not requesting it but for further discussion.)

fralau commented 3 years ago

@ickc @tarleb I think integrating a converter in panflute would be a good idea!

There may also be an env var overriding this behavior, e.g. when pandoc is updated and panflute is yet to catch up then this env var can be used to override the assumed AST version.

If I understand well, it means that planflute would be aware of its own AST version, and could auto-convert if it is out of phase (in and out)? That would be a great idea 👍. In that case, it might be nice to signal this, so that the calling programs calling panflute could issue a warning on the console ("self-deprecated")?

Unfortunately, the pandoc-generated files don't have an AST version number explicitly written in a header (that might be a good idea for an enhancement 🤔 )...

@tarleb, from your experience, could a heuristic be devised that determines which AST version is being used in an input file?

jgm commented 3 years ago

the pandoc-generated files don't have an AST version number explicitly written in a header

Yes, they do.

% pandoc -t json
hi
{"blocks":[{"t":"Para","c":[{"t":"Str","c":"hi"}]}],"pandoc-api-version":[1,21],"meta":{}}

fralau commented 3 years ago

@jgm Oh dear me... I looked at the beginning and not the end.

I confirm, API version number is indeed on the json files (but, for some reason, not on the native files?).

After all, who said that header information should be at the beginning? In Haskelland, it has every right to be at the end, or anywhere for that matter. 😄

NMarkgraf commented 3 years ago

I confirm, API version number is indeed on the json files (but, for some reason, not on the native files?).

Why should the AST version number be in the native files?

fralau commented 3 years ago

@NMarkgraf I don't know; why not? For the sake of symmetry (as well as simplicity), I would have made the two formats isomorphic -- that's less code to produce and maintain. The native format is also part of an API, in their own way. But that's only the perception of an outsider, for what it's worth.

ghamerly commented 3 years ago

@fralau earlier you asked about getting set up on a mac... I'm on a mac, with brew. When pandoc went up to 2.10, I happened to be setting up a docker image for pandoc + panflute, and it saved me since I could keep my version at pandoc-2.9. Here's a dockerimage in case that's helpful. It's uses 2.9 (since that's what I need for panflute), but it should be easy enough to modify to what you want, I hope. I've been building with it for the last few months without issues. There are probably more packages here than you need.

FROM ubuntu:20.04

# these are because timezone installation requires interaction by default
ENV DEBIAN_FRONTEND noninteractive
ENV TZ America/Chicago

RUN apt-get update && \
    apt-get install -y \
    python3 python3-pip \
    graphviz python3-pygraphviz libgraphviz-dev \
    texlive-latex-base texlive-latex-recommended texlive-latex-extra texlive-science \
    librsvg2-bin inkscape \
    wget

# install a more particular version of pandoc, downloaded from github
RUN wget -O /pandoc-2.9.2.1-1-amd64.deb https://github.com/jgm/pandoc/releases/download/2.9.2.1/pandoc-2.9.2.1-1-amd64.deb
RUN dpkg -i /pandoc-2.9.2.1-1-amd64.deb

RUN pip3 install matplotlib dot2tex panflute numpy

CMD /bin/bash

After building it with image name pandoc-build, I invoke it like this from the mac:

docker run --rm -it -v `pwd`:/work -w /work pandoc-build pandoc ...

adding any arguments at the end for pandoc.

fralau commented 3 years ago

@ghamerly Thanks a lot for this rundown, which will certainly be useful for MacOS users, to fix this issue!

For my part, I went a different route, with a good old monobloc executable for MacOS, downloaded from pandoc's website. Precompiled apps are a simple workaround, in situations of conflictual libraries such as this one: just download the executable, put it somewhere in $PATH, and voilà. Storage space on disks is dirt cheap, so there is no real downside (as long as it does not become a habit).

That being said, I definitely need to learn more about docker, which seems to be all the rage. There must be a good reason 🤔 .

aubertc commented 3 years ago

There is now a way, if I understand correctly, to get the "old table" formatting, using https://github.com/jgm/pandoc/pull/6575 There is a pandoc.utils.from_simple_table to perform the conversion.

jgm commented 3 years ago

Pandoc 2.11 is now out. There are some very minor changes from 2.10 in JSON encoding of table elements (making them more consistent with JSON encoding of other elements). The pandoc.utils.from_simple_table is also there, but it's part of the Lua interface and won't be much use for this project.

dhimmel commented 3 years ago

Pandoc 2.11.1 was released and the feature gap between pandoc 2.9 and the latest version is growing.

Due to family health issues, I am currently on leave from work and not in town, so even if this is important I have almost zero time to spend on updating.

@sergiocorreia sending my best wishes and thanks for letting us know.

Does anyone else have an idea regarding the complexity of changes required to support the updated AST version? I'm trying to judge whether upgrading panflute to work with AST version 1.21+ is something that will happen within a month, year, or never.

@tarleb do you think ast-migrator is a viable solution currently for the ability to pass panflute a v1.20 AST?

Thanks everyone for their helpful input thus far, and sorry I'm not more helpful in this comment.

ickc commented 3 years ago

AST migrator can be used independent of panflute, and I think it is the most suitable path forward if you absolutely need to upgrade your pandoc now.

I wouldn't suggest putting it within panflute as a stop-gap measure as it is calling to an external dependency. But it may be a good long term solution for future compatibility.

Edit: link: https://github.com/pandoc/ast-migrator

I'm not sure the status of it though. It seems that it is not a stable release yet, and certainly no binary release. So may be not "now" as I said.

NMarkgraf commented 3 years ago

Sorry, because I don't compile Haskel code myself, are there any precompiled binaries of the ast-migrator for mac os and/or windows?Or better asked, will there be (in a couple of days) some precompiled binaries?

I would like to test it, but installing an maintaining an other language is to much work for me.

ickc commented 3 years ago

See edit above. It isn't very difficult to compile your own as stack makes it easy. I'm guessing binaries will be released when it is stable. I'll let it's author to clarify.

tarleb commented 3 years ago

Yes, we want to provide pre-compiled binaries. @jgm thankfully offered to sign the binaries, as I don't have signing certificates neither for Windows nor for macOS. The code is stable enough and usable (but very boilerplatey). What remains is for me to build the infrastructure around binary generation. My time is spread thin, but I hope to have this done around this time next week.

jgm commented 3 years ago

I'm wondering whether someone can simply step in and add the needed code to panflute. It seems like more work to create and use ast-migrator than it would be to simply patch panflute. Isn't what's needed some fairly boilerplate definitions for the new table constructors? I would think that someone familiar with Python and able to look at Text.Pandoc.Definition API documentation should be able to contribute the needed code.

ickc commented 3 years ago

These are some preliminary observations:

139 is created before this issue started, which perform CI over multiple pandoc versions. Without this, all PR that doesn't support pandoc>=2.10 will fail. With this, you can see which versions are failing. I suggest whoever wanting to take a look at this to have this PR in your fork.

The key to fix this issue is elements.py. I made #145 and #146 to improve some logics and (subjectively) readability in it. These do not fix this issue. However, IMO it is a better starting point if you want to give a take on this issue.

All these are cherry picked in the https://github.com/ickc/panflute/tree/fork branch.

The core of the issue is in elements.from_json. e.g. there's a new element called ColWidthDefault. You'd need to implement a class ColWidthDefault, then in _res_func, write a function to tell how ColWidthDefault would want to "eat" c. Repeat for all new and modified elements.

Also note that the from_json has a few if-then cases: "Metadata key (legacy)", "Document (new API)", "Standard cases". To implement ColWidthDefault, we're in "Standard cases", the 2 others is for backward compatibility.

Now the question is, do we want to support pandoc 2.10+ while maintaining compatibility with pandoc 2.7-2.9? If so, you need to add one more if-then cases, and for each modified elements you need to keep the old class and write and new ones there.

My suggestion would be to not support pandoc<2.10. For people using older versions of pandoc, they should use a matching version of panflute as well. This is the principle of package managing mentioned above. However, we lack a proper ecosystem to propagate this information (e.g. conda-forge's panflute feedstock does not have this dependency explicitly.)

Then, we are in the situation that we still need ast-migrator. I suggest that we should still implement the dispatch of pandoc API version and call ast-migrator if needed, that would remove a lot of cluttering in the codebase (good for maintaining.)

Do we all agree of this first? If so, we now have 2 todos:

implement dispatch and calling ast-migrator
implement new/modified elements in elements.py

(2) is not trivial (from the limited amount of time I gave today...), you need to translate what you understand from https://hackage.haskell.org/package/pandoc-types-1.22/docs/Text-Pandoc-Definition.html to elements.py, which are not simple constructors as in pandocfilters.

Then the obvious thing to do is implement (1) first, which is needed after (2) is done to support pandoc<2.10 anyway. AFAIK this is very trivial to do, just add a few lines in elements.from_json. (I could be wrong.)

I can't give a timeline to this. #139 is opened on May 14 and is still not merged (no pressure to the maintainer though!) My time is spent, now having admittedly non-essential #145, #146. I may be able to do (2) after ast-migrator has the ecosystem built around it (i.e. binary dist.) For (1), I'll see if I got time around Christmas (but may be I should spend it on family instead...)

Others feel free to build on it if you have the time.

sergiocorreia commented 3 years ago

Thanks for all the great input.

I'm wondering whether someone can simply step in and add the needed code to panflute. It seems like more work to create and use ast-migrator than it would be to simply patch panflute. Isn't what's needed some fairly boilerplate definitions for the new table constructors? I would think that someone familiar with Python and able to look at Text.Pandoc.Definition API documentation should be able to contribute the needed code.

TBH I don't think it's that much work, just that I've been extremely short on time. I also feel that ast-migrator would complicate things a bit too much.

My suggestion would be to not support pandoc<2.10. For people using older versions of pandoc, they should use a matching version of panflute as well. This is the principle of package managing mentioned above. However, we lack a proper ecosystem to propagate this information (e.g. conda-forge's panflute feedstock does not have this dependency explicitly.)

I kinda feel the same way. Otherwise, panflute will be filled with if api_version<2.10 ... lines, which would make coding a pain.

alerque commented 3 years ago

Now the question is, do we want to support pandoc 2.10+ while maintaining compatibility with pandoc 2.7-2.9?

I would say no, its not worth it. Old systems with old Pandoc packages will have (or can make) Panflute packages from the same era. If people are putting together new workflows using new Panflute they most likely will be best served by new Pandoc. I would just make a hard break and say further releases only support new Pandoc and suggest people use an old release if that doesn't work for them. The code base will be more maintainable going forward with less chance for weird bugs and nobody seriously looses out.

ickc commented 3 years ago

Implementing the naïve ast-migrator call I mentioned above should be very simple and just a dozen lines of code. Unfortunately it is more complicated than I initially think—it depends also on the filter, not only panflute. So it should be done on a per filter basis.

A gotcha would be people using rely on panflute-filters metadata to run the filter automatically, then the dispatch happening within the filter would not work.

A better solution will be for filter to have a way to tell panflute what pandoc api version it supports, and panflute handling all the conversions in between.

I think it is still doable. It is going to help those who know nothing about maintaining the whole stack (where in pandoc filters it is a totally manual process—pandoc, panflute, then filter version.) I can give it a shot if we have consensus on it.

NMarkgraf commented 3 years ago

Now the question is, do we want to support pandoc 2.10+ while maintaining compatibility with pandoc 2.7-2.9? If so, you need to add one more if-then cases, and for each modified elements you need to keep the old class and write and new ones there.

If someone would ask me, I would vote for a clean cut. The current panflute is for pandoc 2.7-2.9. The next one should have a new release major number and should be strictly for the new AST (pandoc >= 2.10). If someone want to develop the old stuff for old pandoc stuff, he or she could fork at this point. -- But I don't think that someone will volunteer.

dhimmel commented 3 years ago

The current panflute is for pandoc 2.7-2.9. The next one should have a new release major number and should be strictly for the new AST (pandoc >= 2.10).

I'm in agreement especially given limited development capacity. I would like the filter to check the version and emit a warning or failure when run on unsupported versions. Just to clarify, support should be based on the AST version and not the Pandoc version? Or both?

sergiocorreia commented 3 years ago

The current panflute is for pandoc 2.7-2.9. The next one should have a new release major number and should be strictly for the new AST (pandoc >= 2.10).

Agree with major version bump. We should also drop all the current if pandoc_api_version code sprinkled here and there for support of older pandoc versions

I'm in agreement especially given limited development capacity. I would like the filter to check the version and emit a warning or failure when run on unsupported versions. Just to clarify, support should be based on the AST version and not the Pandoc version? Or both?

It's better to give a warning/error based on pandoc-api-version. Panflute can access that for free when it reads the JSON input, but finding out the Pandoc version is a bit more cumbersome.

tarleb commented 3 years ago

The pandoc version is available through the PANDOC_VERSION environment variable, but checking the API version should usually be enough.

sergiocorreia commented 3 years ago

The pandoc version is available through the PANDOC_VERSION environment variable, but checking the API version should usually be enough.

Thanks! We can always check against API version but show both on the error message.

ickc commented 3 years ago

@tarleb, is all the env var def documented somewhere? I wonder is there's any env car that can pass pandoc arg to the filter as well.

sergiocorreia commented 3 years ago

@tarleb, is all the env var def documented somewhere? I wonder is there's any env car that can pass pandoc arg to the filter as well.

https://pandoc.org/lua-filters.html#global-variables (See the part "Global variables" in case the href id doesn't work)

I think this is it. Need to double check though.

Also, I'm thinking we can just add all these globals as properties of the doc object

ickc commented 3 years ago

Looks like they are in the wrong section then (Lua filter). This is going to be very useful to get the pandoc arg so that filter can work more seamlessly with pandoc.

It seems something very easy to implement. We'll probably discuss it more in a separate issue.

sergiocorreia / panflute

Supporting pandoc 2.11 #142

Note on MacOS

139 is created before this issue started, which perform CI over multiple pandoc versions. Without this, all PR that doesn't support pandoc>=2.10 will fail. With this, you can see which versions are failing. I suggest whoever wanting to take a look at this to have this PR in your fork.