search

tel8618217223380 / gwtwiki

Automatically exported from code.google.com/p/gwtwiki
0 stars 0 forks source link

Recursion counters managed incorrectly #110

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Some recursion counters are nether decremented.

This shows up as not included templates at the end of a big wiki text.

Original issue reported on code.google.com by axelclk@gmail.com on 2 Aug 2012 at 10:50

GoogleCodeExporter commented 9 years ago 
Committed r6252

Original comment by axelclk@gmail.com on 2 Aug 2012 at 12:50

GoogleCodeExporter commented 9 years ago

Original comment by axelclk@gmail.com on 8 Aug 2012 at 1:01

GoogleCodeExporter commented 9 years ago 
There is some confusion about depth level and global recursive call limit.

I noticed that the latest trunk has deprecated the global recursive call limit 
- this had nothing to do with the recursive depth.

I can't reproduce this off top of my head, but when parsing all of the 
wikipedia content in the past, I have run into cases where the recursion depth 
was managed properly (repeated +/- depth counter calls), but the parses was 
still stuck in an infinite loop - stuck at the same depth.

As a result, I proposed this as a final limit on the total number of recursive 
calls allowed for a given content irrespective of the parser depth. It is also 
possible to create an attack where a template could inject a very large list of 
the same template s to be evaluated sucking up all the CPU for a very long 
time. This is something that cannot be guarded against looking at the main 
content length/size alone.

I plan to keep this in my snapshot of your trunk, but it would be safer to 
bring it back. The documentation comments should be corrected to omit reference 
to depth.

Cheers.

Original comment by dfi...@itmatter.com on 10 Aug 2012 at 3:22

GoogleCodeExporter commented 9 years ago 
here is an example of 3.0.19 final running into a recursive loop where 
revisions up to r6251 were not:

Import the contents of the following revisions and pages:
https://en.wikipedia.org/w/index.php?title=United_States&oldid=447703673
https://en.wikipedia.org/w/index.php?title=Template:Pp-semi&oldid=268272168
https://en.wikipedia.org/w/index.php?title=Template:Pp-protected&oldid=388458886
https://en.wikipedia.org/w/index.php?title=Template:Hatnote&oldid=388526872
(the latter was renamed now - you need to use the old title: "Template:Dablink")

This eventually leads to errors like this:
     [java] java.lang.OutOfMemoryError: Java heap space
     [java]     at java.util.Arrays.copyOf(Arrays.java:2894)
     [java]     at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:147)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:188)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:72)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1836)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java] java.lang.OutOfMemoryError: Java heap space
     [java]     at java.util.Arrays.copyOf(Arrays.java:2894)
     [java]     at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:147)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:188)
     [java]     at java.lang.StringBuilder.append(StringBuilder.java:72)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1836)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplateOrTemplateParameter(TemplateParser.java:571)
     [java]     at info.bliki.wiki.filter.TemplateParser.runParser(TemplateParser.java:288)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseRecursive(TemplateParser.java:161)
     [java]     at info.bliki.wiki.model.AbstractWikiModel.substituteTemplateCall(AbstractWikiModel.java:1829)
     [java]     at info.bliki.wiki.filter.TemplateParser.parseTemplate(TemplateParser.java:681)

Original comment by nico.kru...@googlemail.com on 13 Aug 2012 at 9:42

GoogleCodeExporter commented 9 years ago 
interestingly though the old version did not even get a "template recursion 
limit exceeded" message...

Original comment by nico.kru...@googlemail.com on 13 Aug 2012 at 9:53

GoogleCodeExporter commented 9 years ago 
Did you try to increase the HeapMemory for the JVM or lower the 
Configuration.TEMPLATE_RECURSION_LIMIT ?

How much HeapMemory did you use for the test you listed above?

Original comment by axelclk@gmail.com on 21 Aug 2012 at 9:42

GoogleCodeExporter commented 9 years ago 
I did not touch any of these two, they are set to their default value in both 
versions of the bliki engine

I see the tomcat with the new version exploding to over 2GB of RAM before it 
crashes though

Original comment by nico.kru...@googlemail.com on 27 Aug 2012 at 3:16

GoogleCodeExporter commented 9 years ago 
nevermind - the error was in my code and the change fixing something in bliki 
probably triggered it :)

Original comment by nico.kru...@googlemail.com on 28 Sep 2012 at 5:34