Custom Pipes

[Eisfunke]

Currently, only the predefined set of pipes is supported. A way to specify custom pipes through the API might be nice.

This should be accomplishable by identifying a pipe through it's Text name instead of through a predefined Pipe constructor and looking the name up in a map (something along the lines of TemplateTarget a => [(Pipe, Val a -> Val a)]) when applying the pipe instead of through a pattern match directly. Such a map could then be provided through the API if extra pipe functionality is desired.

A downside would be that that would lose some safety, invalid pipes could not be detected during parsing but only on compilation of the template. It would also add some complexity, but not much (I think).

What do you think?

[jgm]

It's worth considering, I'm not sure. What sort of thing would you like to do that you can't currently?

[gDanix]

I would make use of something like this right now, for example, to embed images in base64 in my HTML templates. Also for render a particular bibliography in a choosen place in the document.

I would like to contribute myself a customized pipes implementation, by

1. adding a new type constructor to the Pipe datatype (Probably CustomPipe Text)
2. use this pipe when no other one matches the name (pPipe function, change _ -> fail ..... by name -> CustomPipe name)
3. and in the applyPipe, implement the lookup of the code for the pipe (probably name.hs??) in some directory (probably where other filters are??), compile it, and launch it (it should contain the function Val a -> Val a)

But I don't have much time right now, although if this is not implemented the next year, I'll start implementing it myself :)

Other, dirtiest solution would be to add a "system" pipe, defined in a similar way, that launches a shell command specified, with the input content as parameter, and outputs to the document what the command prints to screen (something kinda like a CGI/FastCGI in web servers). This would be the most flexible solution, because it doesn't enforce any language, or source code structure. In case of error, the error could be outputted to the document, so no changes to the applyPipe function would be necessary

[tilowiklundSensmetry]

I'd be interested in this too, I need to print integers in non-decimal bases (primarily some config formats that use hexadecimal for memory addresses).

[jgm]

compile it, and launch it

This would require that we depend on the ghc library, which would change this from being a slim library to being a very fat one.

add a "system" pipe, defined in a similar way, that launches a shell command specified,

This would work in instances of TemplateMonad that do IO, but not in all of them. I suppose one could add a method to TemplateMonad for running an external command, and implement it as fail for other instances -- but I don't think that's very clean design. Also, the security implications are worth considering. People could use a template (perhaps unknowingly, if it's placed in their user data directory) that runs an arbitrary shell command!

The idea of changing the API to allow a custom pipe that is defined in the calling (Haskell) program seems reasonable. This wouldn't help ordinary pandoc users but it could help people who use pandoc as a library. But I don't see a great way to make it "dynamically loadable."

[jgm]

We could consider adding a default pipe for numeric base conversion.

[gDanix]

Yes, this feature wont be easy to implement.

Also, very good idea pointing to the security implications, I didn't think about it.

But I don't understand quite well what TemplateMonad has to do with this proposal (sorry, I'm very novice to Haskell). What I proposed is to add the new pipe in the Pipe dataclass, and then run the unsafe system call in the applyPipe function. But maybe this boils down to the TemplateMonad typeclass? I don't know...

In Pandoc I think Lua is used as a scripting language to implement filters, but I guess the Lua lib is not used in doctemplates, right?

And also librar-ize the pipes implementations, so that anybody can contribute arbitrarily random pipes in a dedicated repo, that is then included when compiling doctemplates will be cumbersome, too.

I'm out of ideas. You better know, for sure. We can't ask you for more, you are doing an awesome work. I wish I can help someday.

El 15/9/21 a las 18:03, John MacFarlane escribió:

compile it, and launch it

This would require that we depend on the ghc library, which would change this from being a slim library to being a very fat one.

add a "system" pipe, defined in a similar way, that launches a
shell command specified,

This would work in instances of TemplateMonad that do IO, but not in all of them. I suppose one could add a method to TemplateMonad for running an external command, and implement it as |fail| for other instances -- but I don't think that's very clean design. Also, the security implications are worth considering. People could use a template (perhaps unknowingly, if it's placed in their user data directory) that runs an arbitrary shell command!

The idea of changing the API to allow a custom pipe that is defined in the calling (Haskell) program seems reasonable. This wouldn't help ordinary pandoc users but it could help people who use pandoc as a library. But I don't see a great way to make it "dynamically loadable."

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jgm/doctemplates/issues/11#issuecomment-920152943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIQ3WNU2WXJY6ADNNKIKSDUCC7VTANCNFSM4RTXLZPQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[exodrifter]

I'm using doctemplates as a library in my own project and having custom pipes is something I'd like to use. In particular, I'm using doctemplates to generate a static website and it would be nice to convert some zoned timestamp into the GMT time zone:

PostedAt: ${it.timestamp/toGMT}

I don't think it would make sense for doctemplates to deal with time shenanigans.

[jgm]

Seems like there's a lot of interest in this. Is anyone interested in making a PR so we could consider it more concretely?