Type of extension required for dynamic content injection using place holders

[mattadamson]

Related to

https://github.com/Swagger2Markup/swagger2markup/issues/142

We'd like to create some kind of extension which allows syntax such as

{{SQL='SELECT firstCol, secondCol FROM MyModelDefnition'',Model=Contact',Cols='First column', ;'Second column'}}

So within the extension the content within {{ and }} is replaced with code logic e.g. query to a database to build a ascii doc formatted table. Then when finished the token is completely replaced with this custom mark up.

From the many extensions supported which is most appropriate for this as I'm not clear?

Thanks

[mojavelinux]

If the syntax you're proposing can appear on the same line as other text, then you most likely want to use an InlineMacro with a custom pattern. You can see an example in Ruby here:

https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/mentions-inline-macro.rb

You are going to require at least one blank space inside the curly braces. Otherwise, you run a (small) risk of the expression getting interpreted as an attribute reference.

If the expression cannot appear on the same line as other text, then I recommend instead using a block:

[query]
SQL='SELECT firstCol, secondCol FROM MyModelDefnition'',Model=Contact',Cols='First column', ;'Second column'

[mattadamson]

thanks @mojavelinux very helpful. Having the content on a separate line or not probably doesn't make much difference in my case however it seems more reusable if it does support both i.e. in place, so will start with that form.

[mattadamson]

Partially related to this issue in the extension I'd like to create a table from dynamic data. In the swagger2markup project we had classes such as MarkUpDocBuilder and methods such as tableWithColumnSpecs to easily build tables. Does anything similar exist in asciidoctorj or supporting common modules?

[mattadamson]

I noticed I can actually create / reuse the class although oddly enough if we use

protected Object process(final AbstractBlock parent, final String target, final Map<String, Object> attributes)
{
    AsciiDocBuilder builder = new AsciiDocBuilder();
    return builder.boldText("mybold").toString();

The actual text content which appears in the generated HTML documented is

mybold

i.e. it's in the ascii doc syntax and not converted to HTML as per the configuration. Does this mean the extensions e.g. inline processor always expect the final output format mark up to be returned?

I wanted to build an extension which could return ascii doctor mark up in place of a custom macro which then be used and converted e.g. mybold converted to a bold tag for HTML

Thanks

[mattadamson]

does anyone else do similar e.g. through an extension change the ascii doc generated mark up before final processing? If so how do you achieve this? Thanks

[lordofthejars]

I think that you can see here an example https://github.com/asciidoctor/asciidoctorj-screenshot

El dt., 3 maig 2016 a les 9:59, mattadamson (notifications@github.com) va escriure:

does anyone else do similar e.g. through an extension change the ascii doc generated mark up before final processing? If so how do you achieve this? Thanks

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/453#issuecomment-216462559

[mojavelinux]

In order to pass on AsciiDoc from an extension, you have to explicitly parse it. Currently, this is hard to do from AsciidoctorJ. I think there's an API in AsciidoctorJ 1.6 to parse children, but I can't remember where we left off that discussion.

Inline macros must always return the converted output. That's because they don't (yet) work like a normal AST node. They are streaming to the output.

[mattadamson]

Thanks that's a great shame. Could it not be possible even if we introduced a new type of extension that simply allowed a string of ascii doc to be returned at a specific extension point placeholder in the adoc file? Ie not requiring full parsing by the engine

[robertpanzer]

On the 1.6.0 you can simply insert Asciidoctor content via Processor.parseContent(parent, lines): https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/asciidoctorj-core/src/main/java/org/asciidoctor/extension/Processor.java#L365

So injecting Asciidoctor content should be possible for BlockProcessors, BlockMacroProcessors and Treeprocessors.

Cheers Robert

[mojavelinux]

On the 1.6.0 you can simply insert Asciidoctor content via Processor.parseContent(parent, lines)

There you go! I knew it was there, I just couldn't remember which class it was on.

[mojavelinux]

Could it not be possible even if we introduced a new type of extension that simply allowed a string of ascii doc to be returned at a specific extension point placeholder in the adoc file?

Any of that would require a change to core and how extension are run. AsciidoctorJ can add a lot of "porcelain" on top, but it can't (easily) change how the processor actually works.

[mojavelinux]

So injecting Asciidoctor content should be possible for BlockProcessors, BlockMacroProcessors and Treeprocessors.

Exactly. As Robert points out, what you are doing is returning additional nodes in the AST. Thus, it's important to remember that these extension points augment the AST. They do not augment the AsciiDoc raw text. The only extension that can do that is a Preprocessor. The downside of a Preprocessor is that it doesn't have any structural context. It just seems a stream of characters, nothing more.

[mojavelinux]

Having said that, the Processor.parseContent(parent, lines) API gives you the opportunity to parse AsciiDoc raw text, so you can still generate a bunch of it. You just have to parse it when you are ready to return from the extension point.

[mattadamson]

thanks all. appreciate the detailed feedback. However I'm not clear if any / all of this possible in the current 1.5.4 release. We actually use ascii doctor in combination with swagger2mark up so upgrading to 1.6 may cause other issues. We also use ascii doctor pdf so perhaps this also needs to be upgraded?

I gather we should switch from an inline macro processor to a block processor. You mention a preprocessor would this still allow us to substitute some text block e.g. where

{{SQL='SELECT firstCol, secondCol FROM MyModelDefnition'',Model=Contact',Cols='First column', ;'Second column'}}

is referenced we could extract / parse the attributes and then return an ascii doc string representing this table it's just we'd have to manually create the text string with ascii doc markup? I certainly don't mind doing that for now as a table is relatively simple. I presume we could also reuse this module to help build the mark up string

https://github.com/Swagger2Markup/markup-document-builder

Thanks

[mattadamson]

These are the pom dependencies I currently specify

<groupId>org.asciidoctor</groupId>
<artifactId>asciidoctor-maven-plugin</artifactId>
<version>1.5.3</version>
<dependencies>
    <dependency> 
        <groupId>org.asciidoctor</groupId>
        <artifactId>asciidoctorj</artifactId>
        <version>1.5.4</version>
    </dependency>
    <dependency> 
        <groupId>org.asciidoctor</groupId>
        <artifactId>asciidoctorj-pdf</artifactId>
        <version>1.5.0-alpha.11</version>
    </dependency>
    <dependency> 
        <groupId>org.jruby</groupId>
        <artifactId>jruby-complete</artifactId>
        <version>1.7.21</version>
    </dependency>

I then noticed errors such as

Caused by: org.apache.maven.plugin.PluginContainerException: An API incompatibil ity was encountered while executing org.asciidoctor:asciidoctor-maven-plugin:1.5 .3:process-asciidoc: java.lang.NoSuchMethodError: org.asciidoctor.internal.JRuby RuntimeContext.get()Lorg/jruby/Ruby;

Does anyone have a working example pom.xml referencing the maven plug in and ascii doctor j 1.6 they can share? I presume this is the only way I can get a preprocessor working to generate ascii doc markup

[mattadamson]

I also tried creating a pre processor within the current version by adding the registry and then defining as

public class AsciiDoctorPreProcessor extends Preprocessor { public AsciiDoctorPreProcessor(String macroName, Map<String, Object> config) { super(config); } @Override public PreprocessorReader process(Document document, PreprocessorReader reader) { // TODO Auto-generated method stub return null; } }

We see an odd exception before the breakpoint in process is reached

Failed to load AsciiDoc document - wrong number of arguments (1 for 2) at org.asciidoctor.internal.JRubyAsciidoctor.renderFile(JRubyAsciidoctor .java:345) at org.asciidoctor.maven.AsciidoctorMojo.renderFile(AsciidoctorMojo.java :289) at org.asciidoctor.maven.AsciidoctorMojo.execute(AsciidoctorMojo.java:18 5) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(Default BuildPluginManager.java:106) ... 20 more Caused by: org.jruby.exceptions.RaiseException: (ArgumentError) asciidoctor: FAI LED: swagger-doc-generator/src/docs/ asciidoc/indexadoc: Failed to load AsciiDoc document - wrong number of arguments (1 for 2)

[mojavelinux]

I gather we should switch from an inline macro processor to a block processor. You mention a preprocessor would this still allow us to substitute some text block

Here are my recommendations, take them or leave them.

1. Using a block processor is preferred because it's the easiest type of extension to write and has the least chance of causing side effects.
2. I don't recommend using a preprocessor, especially for this use case. A preprocessor can lead to a lot of other side effects and is difficult to design well. It's really only something you should use for adding preprocessor directives that have no awareness of context.
3. You could develop an inline macro if you choose something other than double curved brackets. For instance, you could use the syntax {[QUERY]}. Then, the syntax won't get caught up in the attribute substitution.

If you can get (3) to work, I think that's your best bet. If that fails, then drop back to (1).

[mattadamson]

Continuing on from issue 392 and Robert's response

Hi Matt,

a BlockProcessor could look like this, it will add two paragraphs, one with the content Hello and a second with the content World:

@Name("myblockname") @Contexts(Contexts.CONTEXT_PARAGRAPH) @ContentModel(ContentModel.SIMPLE) public class MyBlockProcessor extends BlockProcessor { @Override public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) { Block block = createBlock(parent, "open", (String) null); parseContent(block, Arrays.asList( "Hello", "", "World")); return block; } } The parent will be the section of the block for which the BlockProcessor is called, so it is really the parent and not the block being processed. To get the content of the block to process you can call the reader. If you don't care for that you can keep the reader untouched.

Your BlockProcessor has to create and return the block that should be rendered instead of the processed block. So you create an open block, because this one can have child blocks. As content you pass null as you want to add parsed Asciidoctor content via the following call of parseContent.

Just one small remark about your issues: Could you please fence your source and the stack traces with three backquotes? It is incredibly hard to read if the content is not rendered monospaced.

Additionally it would be awesome if you could open new issues, as I think we're now at a point where your issue has no relation to the original issue.

Cheers Robert

Thanks Robert. perhaps I'm misunderstanding however I thought given that we could simple pass some generic ascii doctor text in places of the paragraph content e.g. this string

ID	Name
firstcol	secondcol

Block block = createBlock(parent, "open", (String) null);
parseContent(block, Arrays.asList( tableAsciiDocMarkup));
return block;

[robertpanzer]

I think you got the syntax for tables wrong. This runs like a train for me:

    @Override
    public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) {
        Block block = createBlock(parent, "open", (String) null);
        parseContent(block,
            Arrays.asList(
                "|===",
                "|ID|Name",
                "",
                "|firstcol|secondcol",
                "|==="));
        return block;
    }

[mattadamson]

thanks that does indeed work perfectly. I was actually using what I thought was complete table mark up from the MarkUpDocBuilder instance shown below

        MarkupDocBuilder builder = new MarkdownBuilder();
        final ArrayList<List<String>> rowColumns = new ArrayList<>();
        final ArrayList<String> rowCells = new ArrayList<>();
        rowCells.add("firstcol");
        rowCells.add("secondcol");
        rowColumns.add(rowCells);
        List<MarkupTableColumn> columnSpecs = new ArrayList<MarkupTableColumn>();
        columnSpecs.add(new MarkupTableColumn("ID"));
        columnSpecs.add(new MarkupTableColumn("Name"));
        builder.tableWithColumnSpecs(columnSpecs, rowColumns);
        tableAsciiDocMarkup = builder.toString();

This produced the same output without the leading |=== and trailing |=== however that's fine I can follow up on a different issue for the other project this class is part of.

Many thanks for all your help here. We might want to include this simple example in the extensions section somewhere as parseContent is so useful. We can close this issue now

[mattadamson]

I also realised the issue with the above is simply changing the instance from MarkdownBuilder to AsciiDocBuilder produces the correct formatted mark up :+1:

[mattadamson]

I've continued developing this and whilst the single table combined mark up seemed to work fine I have issues if I create a couple of tables with line breaks in between e.g. in this code

        public StructuralNode nextBlock() {
            if (!reader.hasMoreLines()) {
                return null;
            }
            IRubyObject nextBlock = getRubyProperty("next_block", reader, ((StructuralNodeImpl)parent).getRubyObject());
            if (nextBlock.isNil()) {
                return null;
            } else {
                return (StructuralNode) NodeConverter.createASTNode(nextBlock);
            }
        }

We see the nextBlock.isNil call evaluate to true and the block simply isn't parsed and therefore in the parseContent call

    public void parseContent(StructuralNode parent, List<String> lines) {
        Ruby runtime = JRubyRuntimeContext.get(parent);
        Parser parser = new Parser(runtime, parent, ReaderImpl.createReader(runtime, lines));

        StructuralNode nextBlock = parser.nextBlock();
        while (nextBlock != null) {
            parent.append(nextBlock);
            nextBlock = parser.nextBlock();
        }
    }

parent.append isn't called. What does isNil really mean here?

[robertpanzer]

Hi Matt,

What null is in Java is Nil in Ruby. So IsNil() returns true if the IRubyObject on which the method is called is Nil/null. (Sounds strange, but that's how JRuby integrates Ruby in Java)

Effectively, and if there is no other error that I might have missed this should be a loop that consumes blocks from the parser until it returns nothing/nil/null.

It would be good if you could post your example somewhere so that we have a way to reproduce the problem.

Cheers Robert

Am Montag, 16. Mai 2016 schrieb mattadamson :

I've continued developing this and whilst the single table combined mark up seemed to work fine I have issues if I create a couple of tables with line breaks in between e.g. in this code

    public StructuralNode nextBlock() {
        if (!reader.hasMoreLines()) {
            return null;
        }
        IRubyObject nextBlock = getRubyProperty("next_block", reader, ((StructuralNodeImpl)parent).getRubyObject());
        if (nextBlock.isNil()) {
            return null;
        } else {
            return (StructuralNode) NodeConverter.createASTNode(nextBlock);
        }
    }

We see the nextBlock.isNil call evaluate to true and the block simply isn't parsed and therefore in the parseContent call

public void parseContent(StructuralNode parent, List<String> lines) {
    Ruby runtime = JRubyRuntimeContext.get(parent);
    Parser parser = new Parser(runtime, parent, ReaderImpl.createReader(runtime, lines));

    StructuralNode nextBlock = parser.nextBlock();
    while (nextBlock != null) {
        parent.append(nextBlock);
        nextBlock = parser.nextBlock();
    }
}

parent.append isn't called. What does isNil really mean here?

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/453#issuecomment-219470092

[mattadamson]

thanks @robertpanzer , perhaps my ascii doc was wrong however I'm trying to do similar to the AsciiDocBuilder class which with the earlier example maybe similar to

          parseContent(block,
              Arrays.asList(
                  ".My table",
                  "|===",
                  "|ID|Name",
                  "",
                  "|firstcol|secondcol",
                  "|==="));

It seems including ".My table" as a table header title doesn't display or rather doesn't find any content in the code walk through above

although looking at all other examples this seems to be the correct syntax for ascii doc table headings

[robertpanzer]

Matt,

I just tried your Asciidoctor example, that means I added a .My table at the beginning of the text passed to parseContent() and it seems to create the expected result. That means from this:

    @Override
    public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) {
        Block block = createBlock(parent, "open", (String) null);
        parseContent(block,
            Arrays.asList(
                ".My table",
                "|===",
                "|ID|Name",
                "",
                "|firstcol|secondcol",
                "|==="));
        return block;
    }

I get this:

<div class="content">
<table class="tableblock frame-all grid-all spread">
<caption class="title">Table 1. My table</caption>
<colgroup>
<col style="width: 50%;">
<col style="width: 50%;">
</colgroup>
<thead>
<tr>
<th class="tableblock halign-left valign-top">ID</th>
<th class="tableblock halign-left valign-top">Name</th>
</tr>
</thead>
<tbody>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock">firstcol</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">secondcol</p></td>
</tr>
</tbody>
</table>
</div>

This looks perfectly fine for me.

How is this different from your result?

Cheers Robert

[mattadamson]

thanks @robertpanzer this works fine now through the AsciiDocMarkupBuilder

    parseContent(block, Arrays.asList(builder.toString().split("\r\n")));

I've run into an odd issue where another environment is outputing the raw ascii doc tags when generating HTML and on my local system it works fine. I'm not sure why although the maven modules / dependencies are the same.

[mattadamson]

Does anyone have any thoughts on what we may see this issue? It seems to be present on UNIX systems eg a Mac and Linux OS however on Windows the table is generated fine. Could it be a case sensitive issue somewhere in the library? Do others use on UNIX to generate tables through processors and convert to HTML ok?

Thanks

[mojavelinux]

The split seems fishy. You are splitting on a Windows endline. I always recommend working with Unix endlines when generating and parsing code.

Internally, Asciidoctor converts all endlines to Unix endlines in the original source before it starts parsing to avoid these sorts of issues.

[mattadamson]

thanks @mojavelinux that was the issue. I also found the System.lineSeparator function in Java 1.7 is better than the older form System.get("line.separator")

[mojavelinux]

:+1: