Allow IncludeProcessors to include other files

[eriwen]

Context

I'm attempting to reduce the syntax needed to declare samples through extending Asciidoctor.

My current attempt uses an IncludeProcessor and looks a bit like this:

include::sample[dir="{samplesPath}/path/to/dir",files="file.txt[tag=foo];other_file.txt[]"]

I have written an IncludeProcessor using AsciidoctorJ that doesn't quite work.

I could not find an API that allows me to tell Asciidoctor to process this include. I have not yet tried to write a Preprocessor extension that does this.

Expected Behavior

I would like the file {samplesPath}/path/to/dir/file.txt[tag=foo] to be included, even if I have to call an API to make this happen.

Alternatively, if this a limitation of the library, it'd be cool to have this mentioned in the docs.

Current Behavior

Literally include::{samplesPath}/path/to/dir/file.txt[tag=foo] is written to the rendered output.

[mojavelinux]

To intercept handling of the include directive, you'd use an IncludeProcessor (https://github.com/asciidoctor/asciidoctorj/blob/v1.6.0-alpha.7/docs/integrator-guide.adoc#include-processors).

However, I think for including samples, a better choice is a block macro processor (https://github.com/asciidoctor/asciidoctorj/blob/v1.6.0-alpha.7/docs/integrator-guide.adoc#block-macro-processors). An include processor does not have access to the structure of the document (it doesn't know its surroundings because it's a line preprocessor), but a block macro does.

[mojavelinux]

A block macro can contribute an AST node to the tree, which is exactly what you want to do here.

[robertpanzer]

A couple of years I did this example: https://github.com/robertpanzer/asciidoctorj-extensions-lab/blob/master/source-block-macro/README.adoc I am not sure if it still works, but maybe it helps to show one way how it could be done.

[eriwen]

@mojavelinux @robertpanzer dang you're laser quick responding — 🍻 to that.

I'll check out a block macro here using inspiration from Robert (thanks). I didn't know that a block macro could contribute an AST node here. For some reason I thought that include::s are processed first and so I thought I'd have to use an IncludeProcessor or Preprocessor.

@mojavelinux note that I'm trying to intercept handling of include:: using an IncludeProcessor as you suggest in my failing example. The trouble is including other stuff while processing the include :)

Thanks again for the guidance here. I hope the stuff will be interesting to you once it's ready.

[mojavelinux]

:beers:

Since the include processor is just a line processor, how it interacts with the parsed document isn't always dependable. It depends on how far along the parser has gotten. While a preprocessor directive does happen before the line is fully read, it operates just in time.

If you wanted to do what you were doing, you have to contribute the lines back to the preprocessor reader. This has come several times before and it can be a bit tricky in AsciidoctorJ. I don't remember what the solution was exactly, but somewhere in the issue tracker we've talked about how to get the lines to be processed again.

Regardless, the block macro is still a better solution for this scenario, as Robert's prototype demonstrates. In fact, I'd really like to get that macro standardized so that it's something we can rely on in AsciidoctorJ and perhaps even Asciidoctor itself.

[mojavelinux]

While a preprocessor directive does happen before the line is fully read, it operates just in time.

This is why includes are best for either stitching whole documents together or injecting content inside a delimited block.

[mojavelinux]

Keep in mind that a block macro should only create a pass block as a last resort. You almost always want to create a proper structural node, such as a listing block.

[eriwen]

I see that @robertpanzer's experimental JavaSourceBlockMacroProcessor also requires that one implement the file inclusion and tag filtering.

Is there a way I could invoke Asciidoctor APIs to handle this? I would rather not re-implement indent and other attributes on top of all the file inclusion/filtering that's already going on.

[mojavelinux]

Not currently. That method is not yet part of the public API. However, this is being discussed upstream.

See https://github.com/asciidoctor/asciidoctor/issues/571

[mojavelinux]

For reference, I had to reimplement the tag filtering in Antora (see https://gitlab.com/antora/antora/blob/master/packages/asciidoc-loader/lib/include/include-processor.js). So it's pretty clear it would be nice if an include processor (or even some other extension) could reuse this logic.

[eriwen]

Got it. Thanks for the info.

[mojavelinux]

I could not find an API that allows me to tell Asciidoctor to process this include.

To address $subject directly, when you push an include, the file argument needs to end with .adoc. That's how Asciidoctor knows to process those lines using the AsciiDoc preprocessor (and thus handle any nested includes). You can see this logic here: https://github.com/asciidoctor/asciidoctor/blob/b794da8ea2169e2ea24c3adb14b6068b06aa1794/lib/asciidoctor/reader.rb#L683-L686

There is an open issue to specify that lines should be preprocessed without the file argument ending in .adoc. See https://github.com/asciidoctor/asciidoctor/issues/848#issuecomment-168080073 But setting the file extension of the file argument to .adoc gets the job done for now.