Adding empty Treeprocessor kills TOC

[antoinesd]

I'm starting to do some test with Treeprocessor in version 1.5.2. My class looks like

public class TckMapperTreeProcessor extends Treeprocessor{

    public TckMapperTreeProcessor(Map<String, Object> config) {
        super(config);
    }

    @Override
    public Document process(Document document) {
        List<AbstractBlock> blocks=document.blocks();
        for (AbstractBlock block : blocks) {
            for (AbstractBlock block2 : block.blocks()) {
                if(block2 instanceof Section)
                System.out.println(((Section)block2).id());
            }
        }
        return document;
    }
}

I'm using maven, so I added my extension as dependency. Everything works as exepcted (I have the list of section id in the output) except that the generated document has now an empty table of content. The TOC comes back if I remove the extension dependency.

[mojavelinux]

If I remember correctly from my own experience, this is one of those problems that manifests with AST traversal in AsciidoctorJ 1.5.2. Touching the AST seems to corrupt it in certain ways. If you are doing any extensions at all these days, you should be using AsciidoctorJ 1.6.0 (https://github.com/asciidoctor/asciidoctorj/tree/asciidoctorj-1.6.0#using-a-snapshot-version). Currently it's just a snapshot, but alpha is coming :soon:

[mojavelinux]

Of course, if you still get this with AsciidoctorJ 1.6.0, then we definitely have an issue to address.

[antoinesd]

That's ok Dan, I only wanted to be sure I didn't miss something before going to 1.6.0. As it's more for internal use, I can live with a snapshot.

Le sam. 31 oct. 2015 à 05:20, Dan Allen notifications@github.com a écrit :

Of course, if you still get this with AsciidoctorJ 1.6.0, then we definitely have an issue to address.

— Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/392#issuecomment-152691482 .

[antoinesd]

This said, I am now in a jar hell between asciidoctor maven plugin and my tree processor not using the same JRuby version... So I'll probably have to wait... Unless my initial bug could be fixed in Asciidoctorj 1.5.3 ;)

[mojavelinux]

It should be possible to set the JRuby version used by the Maven plugin. See https://github.com/asciidoctor/asciidoctor-maven-examples/blob/master/asciidoc-to-html-example/pom.xml#L26-L30

[antoinesd]

Yes, I did that. In fact the problem is on Asciidoctorj version. I upgraded my plugin to 1.6.0-SNAPSHOT while Maven plugin still use 1.5.2 (including the snapshot version) when I upgrade Asciidoctorj to 1.6.0-SNAPSHOT my doc generation fails with:

process-asciidoc failed: An API incompatibility was encountered while executing org.asciidoctor:asciidoctor-maven-plugin:1.5.3-SNAPSHOT:process-asciidoc: java.lang.NoSuchMethodError: org.asciidoctor.internal.JRubyRuntimeContext.get()Lorg/jruby/Ruby;

and when I revert to default asciidoctorj version I have:

process-asciidoc failed: An API incompatibility was encountered while executing org.asciidoctor:asciidoctor-maven-plugin:1.5.3-SNAPSHOT:process-asciidoc: java.lang.IncompatibleClassChangeError: Found class org.asciidoctor.ast.Document, but interface was expected

Looks like I'm trapped until Maven plugin supports Asciidoctorj version 1.6.0...

[robertpanzer]

I could reproduce the problem with AsciidoctorJ-1.5.2. Not very helpful but at least I can tell that this problem no longer occurs with the current 1.6.0 and I also added a test case: https://github.com/asciidoctor/asciidoctorj/commit/15941864277815e51ad5d80e34ddf0b077f34b87

[mojavelinux]

Hero! Thanks Robert!

[antoinesd]

Thanks Robert. I'm feeling less alone ;).

[mattadamson]

Related to @antoinesd s earlier comment I'm also getting the error using the maven plug in

Caused by: org.apache.maven.plugin.PluginContainerException: An API incompatibil
ity was encountered while executing org.asciidoctor:asciidoctor-maven-plugin:1.5
.3:process-asciidoc: java.lang.NoSuchMethodError: org.asciidoctor.internal.JRuby
RuntimeContext.get()Lorg/jruby/Ruby;

and asciidoctorj 1.6.0-SNAPSHOT. Is this supposed to be fixed or a known workaround? I didn't want to open a new issue and following up this issue as similar. It relates to exploring a solution using 1.6.0-SNAPSHOT for https://github.com/asciidoctor/asciidoctorj/issues/453

[mojavelinux]

Currently, the Maven plugin is not compatible with AsciidoctorJ 1.6.0. The Maven plugin was using an internal API which has changed.

The 1.5.4 release of the Maven plugin should handle this by doing runtime detection of the API so that it can work with both versions.

Here's the line in question:

https://github.com/asciidoctor/asciidoctor-maven-plugin/blob/asciidoctor-maven-plugin-1.5.3/src/main/java/org/asciidoctor/maven/AsciidoctorMojo.java#L245

[mattadamson]

thanks @mojavelinux , If I switch from 1.5.3 to 1.5.4 I get this error

    at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.maven.wagon.TransferFailedException: Failed to transfer fi le: http://oss.jfrog.org/artifactory/oss-snapshot-local/org/asciidoctor/asciidoc tor-maven-plugin/1.5.4/asciidoctor-maven-plugin-1.5.4.pom. Return code is: 409 , ReasonPhrase:Conflict.

[robertpanzer]

Hi Matt,

Afaik there's no dedicated version 1.5.4 of the maven plugin. To use asciidoctor 1.5.4 within a Maven build use the 1.5.3 plugin and add a dependency on asciidoctorj 1.5.4 to the plugin configuration.

Cheers Robert

Am Samstag, 7. Mai 2016 schrieb mattadamson :

thanks @mojavelinux https://github.com/mojavelinux , If I switch from 1.5.3 to 1.5.4 I get this error

at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.maven.wagon.TransferFailedException: Failed to transfer fi le: http://oss.jfrog.org/artifactory/oss-snapshot-local/org/asciidoctor/asciidoc tor-maven-plugin/1.5.4/asciidoctor-maven-plugin-1.5.4.pom. Return code is: 409 , ReasonPhrase:Conflict.

— You are receiving this because you were assigned. Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/392#issuecomment-217655764

[mattadamson]

thanks @robertpanzer , that probably won't help me here though as I'm trying to find a solution using ascii doctor 1.6 to explore the solution in

https://github.com/asciidoctor/asciidoctorj/issues/453

[mojavelinux]

My point is, we need to a) fix the Maven plugin so it works with AsciidoctorJ 1.6.0 and b) release 1.5.4 ;)

[robertpanzer]

Will try to create a PR tmw. Have 11 hours of car driving in the last 36 hours in my bones, so need some time to recover;)

Am Samstag, 7. Mai 2016 schrieb Dan Allen :

My point is, we need to a) fix the Maven plugin so it works with AsciidoctorJ 1.6.0 and b) release 1.5.4 ;)

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/392#issuecomment-217656623

[mattadamson]

:) sure. If I can help in anyway to fix the plug in or release 1.5.4 let me know too, thanks.

[robertpanzer]

If you like to do it this would be great and more than welcome!!! get() has been replaced by get(Asciidoctor) on the same class. The solution should detect by reflection which method exists and invoke it.

This is not nice but I think to be compatible with both 1.5.x and 1.6.0 this is necessary.

Cheers Robert

Am Samstag, 7. Mai 2016 schrieb mattadamson :

:) sure. If I can help in anyway to fix the plug in or release 1.5.4 let me know too, thanks.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/asciidoctor/asciidoctorj/issues/392#issuecomment-217656773

[mattadamson]

I don't understand the code 100% however I've tried in the snippet below. Note it's rather long due to the many catch clauses and saw the method contract expects to throw only MojoExecutionException. We also can't use JDK 1.7 multiple try clauses in one block

`// public static Ruby get(Asciidoctor asciidoctor) {

// JRubyRuntimeContext.get(asciidoctor); Ruby rubyInstance = null; try { rubyInstance = (Ruby) JRubyRuntimeContext.class.getMethod("get").invoke(null); } catch (NoSuchMethodException e) { if(rubyInstance == null) { try { rubyInstance = (Ruby) JRubyRuntimeContext.class.getMethod("get", Asciidoctor.class).invoke(null, asciidoctor); } catch (NoSuchMethodException e1) { throw new MojoExecutionException("Failed to invoke get(AsciiDoctor) for JRubyRuntimeContext",e); } catch (SecurityException e1) { throw new MojoExecutionException("Failed to invoke get(AsciiDoctor) for JRubyRuntimeContext",e); } catch (IllegalAccessException e1) { throw new MojoExecutionException("Failed to invoke get(AsciiDoctor) for JRubyRuntimeContext",e); } catch (IllegalArgumentException e1) { throw new MojoExecutionException("Failed to invoke get(AsciiDoctor) for JRubyRuntimeContext",e); } catch (InvocationTargetException e1) { throw new MojoExecutionException("Failed to invoke get(AsciiDoctor) for JRubyRuntimeContext",e); }

        }
    }
    catch (SecurityException e)
    {
        throw new MojoExecutionException("Failed to retrieve get method information from JRubyRuntimeContext",e);
    }
    catch (IllegalAccessException e)
    {
        throw new MojoExecutionException("Failed to invoke get for JRubyRuntimeContext",e);
    }
    catch (IllegalArgumentException e)
    {
        throw new MojoExecutionException("Failed to invoke get for JRubyRuntimeContext",e);
    }
    catch (InvocationTargetException e)
    {
        throw new MojoExecutionException("Failed to invoke get for JRubyRuntimeContext",e);
    }

    String gemHome = rubyInstance.evalScriptlet("ENV['GEM_HOME']").toString();

`

This did seem to compile and pass all tests.

Going back to my original issue if I then referenced this build of the maven plug in and asciidoctor 1.6 we see

Caused by: org.jruby.exceptions.RaiseException: (NoMethodError) asciidoctor: FAILED: d:/index.adoc: Failed to load AsciiDoc document - undefined method `attributes' for #String:0x12c76d6e

    at RUBY.build_block(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoc

tor/parser.rb:1089) at RUBY.next_block(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoct or/parser.rb:883) at RUBY.next_section(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciido ctor/parser.rb:315) at RUBY.parse(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor/pa rser.rb:67) at RUBY.parse(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor/do cument.rb:471) at RUBY.load(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor.rb: 1344) at RUBY.convert(uri:classloader:/gems/asciidoctor-1.5.4/lib/asciidoctor. rb:1462)

although this could be something related to the block processor I'm trying to add and unrelated?

I'm happy to try and create a PR for the above if you think it looks ok?

[robertpanzer]

Do you have your BlockProcessor somewhere on GitHub?

[robertpanzer]

Are you sure your BlockProcessor returns a Block and not a String?

[mattadamson]

without an example I'm not entirely sure what process is expected to return however I assume it was the node being passed. Given what was mentioned on the other issue 453 in relation to parseContent I tried a test calling as below

@Override public Object process(StructuralNode arg0, Reader arg1, Map<String, Object> arg2) { parseContent(arg0, Arrays.asList("*test*")); return arg0; }

Also do you want me to submit the current code in a PR?

[robertpanzer]

Would be great if you could provide a PR.

Regarding the BlockProcessor there's a documentation here: https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/docs/integrator-guide.adoc#block-processors

I'm currently sitting behind a small screen so cannot retest. (Therefore please excuse if I'm to laconic) Can imagine that you have to consume the content of the reader like the example does as well.

[mattadamson]

Sure I created the PR here

https://github.com/asciidoctor/asciidoctor-maven-plugin/pull/218

I'm not really sure how to return a block using the parseContent call e.g. currently I have

` @Override public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) { MarkupDocBuilder builder = new MarkdownBuilder(); final ArrayList<List> rowColumns = new ArrayList<>(); final ArrayList rowCells = new ArrayList<>(); rowCells.add("firstcol"); rowCells.add("secondcol"); rowColumns.add(rowCells); builder.table(rowColumns);

    return createBlock(parent, "paragraph", builder.toString(), attributes);

// parseContent(parent, Arrays.asList(builder.toString())); } `

[mattadamson]

On a related note I now see many other table related methods in 1.6 I never saw before. Perhaps they would be more suitable to build the table than using the other project mark up doc builder from swagger2markup. e.g we see createTable and then access to columns

[robertpanzer]

Yes, AST support for tables has been added to the 1.6.0 branch and is not available with 1.5.x. If you want to use the Asciidoctor parser to create a table from an Asciidoctor string or create the AST nodes yourself is just a matter of taste.

You can see an example of a processor that invokes parseContent() here: https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/asciidoctorj-core/src/test/groovy/org/asciidoctor/extension/WhenAnExtensionAppendsChildBlocks.groovy#L82

(parseContent is also new as of 1.6.0) It is written in Groovy, but should be easily transferrable to Java.

[mattadamson]

Thanks Robert and what type should I return from the proceed method? I'm not clear from the groovy example

On 8 May 2016, at 11:21, Robert Panzer notifications@github.com wrote:

Yes, AST support for tables has been added to the 1.6.0 branch and is not available with 1.5.x. If you want to use the Asciidoctor parser to create a table from an Asciidoctor string or create the AST nodes yourself is just a matter of taste.

You can see an example of a processor that invokes parseContent() here: https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/asciidoctorj-core/src/test/groovy/org/asciidoctor/extension/WhenAnExtensionAppendsChildBlocks.groovy#L82

(parseContent is also new as of 1.6.0) It is written in Groovy, but should be easily transferrable to Java.

— You are receiving this because you commented. Reply to this email directly or view it on GitHub

[robertpanzer]

Hi Matt,

I hope that the integrator guide is able to answer this question. We put quite some effort into it: https://github.com/asciidoctor/asciidoctorj/blob/asciidoctorj-1.6.0/docs/integrator-guide.adoc#block-processors

Cheers Robert

[mattadamson]

thanks Robert.

Looking at the groovy example it looks like the StructuralNode is returned i.e. parseContent should append the block so simply returning the parent passed in should suffice.

` public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) { parseContent(parent, Arrays.asList("test"));

    return parent;
}

`

After trying this I received this exception.

5.4/lib/asciidoctor/converter/base.rb:34) at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMe thod.java:101) at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMe thod.java:129) at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIR Method.java:181) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.j ava:197) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java: 161) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.invokeOther9:convert(uri:classloader:/gems/asciidoctor-1.5 .4/lib/asciidoctor/abstract_block.rb) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.RUBY$method$convert$0(uri:classloader:/gems/asciidoctor-1. 5.4/lib/asciidoctor/abstract_block.rb:71) at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMe thod.java:116) at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIR Method.java:146) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.j ava:189) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java: 129) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.invokeOther0:convert(uri:classloader:/gems/asciidoctor-1.5 .4/lib/asciidoctor/abstract_block.rb) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.RUBY$block$content$1(uri:classloader:/gems/asciidoctor-1.5 .4/lib/asciidoctor/abstract_block.rb:80) at org.jruby.runtime.CompiledIRBlockBody.commonYieldPath(CompiledIRBlock Body.java:70) at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:152) at org.jruby.runtime.BlockBody.yield(BlockBody.java:78) at org.jruby.runtime.Block.yield(Block.java:147) at org.jruby.RubyArray.collect(RubyArray.java:2286) at org.jruby.RubyArray.map19(RubyArray.java:2300) at org.jruby.RubyArray$INVOKER$i$0$0$map19.call(RubyArray$INVOKER$i$0$0$ map19.gen) at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite. java:139) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java: 145) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.invokeOther3:map(uri:classloader:/gems/asciidoctor-1.5.4/l ib/asciidoctor/abstract_block.rb) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.abstract_block.RUBY$method$content$0(uri:classloader:/gems/asciidoctor-1. 5.4/lib/asciidoctor/abstract_block.rb:80) at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMe thod.java:101) at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIR Method.java:111) at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers. java:903) at org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntim eHelpers.java:896) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.block.invokeSuper101:content(uri:classloader:/gems/asciidoctor-1.5.4/lib/ asciidoctor/block.rb) at uri_3a_classloader3a.gems.asciidoctor_minus_1_dot_5_dot_4.lib.ascii doctor.block.RUBY$method$content$0(uri:classloader:/gems/asciidoctor-1.5.4/lib/a sciidoctor/block.rb:110) at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMe thod.java:116) at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIR Method.java:146) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.j ava:189) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java: 129)

[robertpanzer]

Hi Matt,

a BlockProcessor could look like this, it will add two paragraphs, one with the content Hello and a second with the content World:

@Name("myblockname")
@Contexts(Contexts.CONTEXT_PARAGRAPH)
@ContentModel(ContentModel.SIMPLE)
public class MyBlockProcessor extends BlockProcessor {
    @Override
    public Object process(StructuralNode parent, Reader reader, Map<String, Object> attributes) {
        Block block = createBlock(parent, "open", (String) null);
        parseContent(block, Arrays.asList( "Hello", "", "World"));
        return block;
    }
}

The parent will be the section of the block for which the BlockProcessor is called, so it is really the parent and not the block being processed. To get the content of the block to process you can call the reader. If you don't care for that you can keep the reader untouched.

Your BlockProcessor has to create and return the block that should be rendered instead of the processed block. So you create an open block, because this one can have child blocks. As content you pass null as you want to add parsed Asciidoctor content via the following call of parseContent.

Just one small remark about your issues: Could you please fence your source and the stack traces with three backquotes? It is incredibly hard to read if the content is not rendered monospaced.

Additionally it would be awesome if you could open new issues, as I think we're now at a point where your issue has no relation to the original issue.

Cheers Robert

[mattadamson]

thanks Robert, yes let's continue on the other issue. I only started on this one as someone mentioned an issue with asciidoctorj 1.6 similar to mine however we're past this now.