search

jgm / gitit

A wiki using HAppS, pandoc, and git
GNU General Public License v2.0
2.15k stars 225 forks source link

support (e.g. as plugin) for processing pandoc JSON format (like filters but via network/unix socket) #477

Open jonassmedegaard opened 9 years ago

jonassmedegaard commented 9 years ago

Seems gitit does not support applying Pandoc filters.

I am setting up a workflow similar to http://gitit.net/collaborative%20authoring. Some of my needs for the resulting PDF design can be solved by use of a custom XeLaTeX template, but some need tweaking the text content. I currently use perl regex but now try recode that as Pandoc filters. I then look at gitit for allowing authors to edit content and instantly fetch a resulting draft PDF - but would then need my Pandoc filters applied as part of the export.

If curious, my concrete use is for editing reports at the European Parliament (didn't succeed for http://islandsofresilience.eu/ and now try again for http://euwiki.org/EUT/2nd-edition). My draft messy code is as at http://source.epfsug.biks.dk/?p=eut.git.

jgm commented 9 years ago

What you want is a plugin.

Filters do the same thing in pandoc, but for gitit, it makes more sense to load the code when the server starts.

See the documentation on plugins.

+++ Jonas Smedegaard [Dec 17 14 08:29 ]:

Seems gitit does not support applying Pandoc filters.

I am setting up a workflow similar to http://gitit.net/collaborative%20authoring. Some of my needs for the resulting PDF design can be solved by use of a custom XeLaTeX template, but some need tweaking the text content. I currently use perl regex but now try recode that as Pandoc filters. I then look at gitit for allowing authors to edit content and instantly fetch a resulting draft PDF - but would then need my Pandoc filters applied as part of the export.

If curious, my concrete use is for editing reports at the European Parliament (didn't succeed for http://islandsofresilience.eu/ and now try again for http://euwiki.org/EUT/2nd-edition). My draft messy code is as at http://source.epfsug.biks.dk/?p=eut.git.

Reply to this email directly or view it on GitHub: https://github.com/jgm/gitit/issues/477

jonassmedegaard commented 9 years ago

Makes sense to implement this feature as a plugin, I guess.

If you mean a unique gitit plugin for each filtering need, I am not sure I agree, however: I write my Pandoc filter in Perl. Transforming that into a long-running service processing Pandoc JSON over a network socket is easy - far easier than learning Haskell to rewrite as a gitit plugin. I imagine it is similar for other non-Haskell filter frameworks as well.

jonassmedegaard commented 9 years ago

You mention plugin documentation - I have failed to locate that on my own: Could you pass me a URL, so that I can check what exists instead of making assumption.

jgm commented 9 years ago

It should be here: https://hackage.haskell.org/package/gitit-0.10.6.1/docs/Network-Gitit-Interface.html

I don't know why the docs for this module didn't build on Hackage.

Anyway, if you do 'cabal haddock' in the source directory after building gitit with the -fplugins option, you should find it at dist/doc/html/gitit/Network-Gitit-Interface.html.

Or you can just look directly at the comments in the source code for that module.

+++ Jonas Smedegaard [Dec 17 14 11:20 ]:

You mention plugin documentation - I have failed to locate that on my own: Could you pass me a URL, so that I can check what exists instead of making assumption.

Reply to this email directly or view it on GitHub: https://github.com/jgm/gitit/issues/477#issuecomment-67376829

jgm commented 9 years ago

You could always write a Haskell plugin that just shells out to your perl program, though this might more difficult than just rewriting in Haskell.

+++ Jonas Smedegaard [Dec 17 14 11:13 ]:

Makes sense to implement this feature as a plugin, I guess.

If you mean a unique gitit plugin for each filtering need, I am not sure I agree, however: I write my Pandoc filter in Perl. Transforming that into a long-running service processing Pandoc JSON over a network socket is easy - far easier than learning Haskell to rewrite as a gitit plugin. I imagine it is similar for other non-Haskell filter frameworks as well.

Reply to this email directly or view it on GitHub: https://github.com/jgm/gitit/issues/477#issuecomment-67375792

jonassmedegaard commented 9 years ago

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your perl program, though this might more difficult than just rewriting in Haskell.

That sounds less elegant to me than extending the existing language-agnostic Pandoc JSON serialization interface:

jgm commented 9 years ago

+++ Jonas Smedegaard [Dec 17 14 12:33 ]:

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your perl program, though this might more difficult than just rewriting in Haskell.

That sounds less elegant to me than extending the existing language-agnostic Pandoc JSON serialization interface:

  • Path handed to Pandoc is executed, and JSON is piped through
  • URL handed to gitit is resolved, and JSON is sent through

But using perl and JSON serialization for this is not very elegant in the first place. After all, we have the AST in a nice in-memory pandoc data structure. A proper gitit plugin just modifies this directly -- the shell is not involved at all, and no JSON serialization/deserialization needs to be done. In a web application, this seems a far better approach.

Note also that the pandoc filters interface is part of the pandoc executable, not the pandoc library, which gitit uses.

jonassmedegaard commented 9 years ago

Quoting John MacFarlane (2014-12-17 22:23:09)

+++ Jonas Smedegaard [Dec 17 14 12:33 ]:

Quoting John MacFarlane (2014-12-17 20:36:13)

You could always write a Haskell plugin that just shells out to your perl program, though this might more difficult than just rewriting in Haskell.

That sounds less elegant to me than extending the existing language-agnostic Pandoc JSON serialization interface:

  • Path handed to Pandoc is executed, and JSON is piped through
  • URL handed to gitit is resolved, and JSON is sent through

But using perl and JSON serialization for this is not very elegant in the first place. After all, we have the AST in a nice in-memory pandoc data structure. A proper gitit plugin just modifies this directly -- the shell is not involved at all, and no JSON serialization/deserialization needs to be done. In a web application, this seems a far better approach.

Note also that the pandoc filters interface is part of the pandoc executable, not the pandoc library, which gitit uses.

Sure, learning Haskell and using solely that language is certainly the most elegant.

Seems gitit is not gonna be as friendly towards simple folks like me as Pandoc (the executable) is, so I guess I will simply have to stay away from gitit (as both the alternatives of learning Haskell and paying someone else to write plugins for me are less suitable for me).

Feel free to close this bugreport, if I understand you correctly that you do not find it relevant to support the neds I describe.

jgm commented 9 years ago

+++ Jonas Smedegaard [Dec 17 14 18:14 ]:

Seems gitit is not gonna be as friendly towards simple folks like me as Pandoc (the executable) is, so I guess I will simply have to stay away from gitit (as both the alternatives of learning Haskell and paying someone else to write plugins for me are less suitable for me).

Feel free to close this bugreport, if I understand you correctly that you do not find it relevant to support the neds I describe.

It can be kept open. I can certainly see the use for this. But, because of my other commitments, I've been in minimal maintenance mode for gitit for some time. If someone wants to contribute a patch for this feature, I'd consider it.

jonassmedegaard commented 9 years ago

Quoting John MacFarlane (2014-12-18 05:04:54)

+++ Jonas Smedegaard [Dec 17 14 18:14 ]:

Seems gitit is not gonna be as friendly towards simple folks like me as Pandoc (the executable) is, so I guess I will simply have to stay away from gitit (as both the alternatives of learning Haskell and paying someone else to write plugins for me are less suitable for me).

Feel free to close this bugreport, if I understand you correctly that you do not find it relevant to support the neds I describe.

It can be kept open. I can certainly see the use for this. But, because of my other commitments, I've been in minimal maintenance mode for gitit for some time. If someone wants to contribute a patch for this feature, I'd consider it.

I appreciate that you recognize its value. Fully understandable that it is not of highest priority for you. :-)

Let's hope someone comes along with similar needs and better Haskell skills than me.

RGD2 commented 6 years ago

I get the impression that adding pandoc filter support of any kind to gitit is nontrivial because pandoc is being used as a library, and filter use (especially thirdparty coughpantablescough) may not have been factored into the design of pandoc when used that way. Well, at least the one time the word 'filter' seems to show up on pandoc's API help page is in the context of xss security. Discuss?

I would absolutely love it if someone were to add a way to import-in place another file with some kind of shell filter... this would be enough to support pandoc filters (since the option could be given) but would also allow things like using a bash one-liner to grab a particular figure out of a pdf, for example. Or a png graph as freshly-generated by a gnuplot. Or even that after an ngspice call.

Alternative workaround for now is to have the 'desktop' cloud9 ide installed on the same machine as gitit, and use it to update wikidata there and push commits directly. This also gives a way to run a shell, but it means that generating/updating such content is an extra step or two, and the resulting files are static graphics which gitit will show on the page.

I guess for this issue, one could exploit a git hook in gitit's wikidata repository to process certain source documents automatically when they change, and push a commit with the updated 'display' files. That would have much the same ultimate effect, although with having to decode gits documentation to do so.