Support for custom content format renderers and transformations

[rgov]

There are some kinds of transformations that I'd like to be able to perform on my content that are not possible or easy with shortcodes.

For instance, it is not currently possible to wrap each rendered section of a Markdown document from its # heading to its last paragraph in a <section> tag, or to change how footnotes are rendered, or to embed a Graphviz diagram in my content file and have it rendered to an embedded SVG by invoking dot, etc. These kinds of changes would require patching Hugo.

Now that Hugo has the ability to execute external tools like Asciidoctor, why not make it more flexible so that users can provide their own content format renderers?

1. Hugo publishes a specification on how a renderer is invoked and how it should return rendered content (e.g., the body of the rendered content, and a list of anchors for the table of contents).

2. The user configures how the renderer is invoked, such as: the tool's path; the file extension for the content; whether to perform further shortcode expansion on the output.

(Rather than allow a complex configuration of command line arguments for an existing tool like Asciidoctor, the user would provide an adapter script.)

This extends nicely to supporting content pipelines (unrelated to existing asset pipelines). The default pipeline for .md files might look like this:

1. Parse file content with Goldmark
2. Generate HTML
3. Perform shortcode expansion
4. Output

If I want to customize this pipeline, I could use the same building blocks, and insert an external transformation (or multiple) that modifies the Markdown before it gets rendered by Goldmark, or before it renders to HTML, etc.

[rgov]

Pipelines were discussed in #3207 but the discussion focused on assets (images, etc.) not content.

A similar proposal is to customize how external helpers are invoked to support different output formats (#6089). The proposal in this issue would be able to achieve this as well, if the output steps of the pipeline can be overridden (for example, to add an HTML to LaTeX step).

Plugins have been discussed but the issue brought up is security. Since this is configured entirely by the user, and not by third-party templates or content, it doesn't have the same drawbacks as #796. (Though, if someone wished, they would be able to implement an exec shortcode for themselves using this proposal, by adding a custom transformation.)

Also, by allowing for external transformation tools, they do not need to all be written in Go. This makes it more attractive than true plug-ins for people who are familiar with languages such as JavaScript or Python.

[rigtorp]

This is related to #7765 Add support for generating figures and diagrams.

[stevegt]

@gadams999 mentions a "taint" mode over in #8398. I like this idea. Make it such that shelling out requires saying hugo --unsafe-shellout or somesuch. It may also support at least some of #796 and/or #7765.

Otherwise the only generic compromise I've ever been able to come up with is to wrap the hugo command execution in an external watch-rebuild loop. A shell script runs /usr/bin/inotifywatch to watch for file changes, then runs a Makefile which rebuilds images, other assets, and goldmark-compatible markdown from external sources, and then finally runs hugo.

A taint mode would get rid of the need for this external loop.

[mlois-efimob]

Hi, sorry for the late comment, I had the same problem with content manipulation, for example to add support for some type of embedded content (formulas, diagrams, etc.), converting all to shortcodes really no make any sense for me, After some review I come to the conclusion that a goldmark extension or integration need to be done, but this needs the repo to be cloned modified and build, I think that a better approach that will fit multiple use cases are that this goldmark extensions/integrations can be plugged to hugo. In other similar golang CLI tools a approach similar to hasicorp plugins is used will lots of success. This will allow that user can add their custom formats to markdown processing without touching hugo build. Sorry in advance if I missing some current or planned Hugo feature for this, I was not aware of the latest news.

[BBaoVanC]

I would love this feature, one example I can think of is making a markdown -> Gopher renderer, so I could make a Gopher version of my site automatically rendered from my markdown source.