Bug in com.sun.java.help.search.BitSet.java leads to a long trail of exceptions

Our organization uses JavaHelp for packaging our help documentation for our product. We've noticed that at some point when a very large file was being indexed via jhindexer, a number of IOExceptions were generated:

java.io.IOException: Can't store Document at com.sun.java.help.search.DefaultIndexerKit.storeToken(DefaultIndexerKit.java:164) at com.sun.java.help.search.DefaultIndexerKit.parseIntoTokens(DefaultIndexerKit.java:117) at com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.addContent(HTMLIndexerKit.java:1092) at com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.addContent(HTMLIndexerKit.java:1064) at com.sun.java.help.search.HTMLIndexerKit$HTMLParserCallback.handleText(HTMLIndexerKit.java:411) at javax.swing.text.html.parser.DocumentParser.handleText(DocumentParser.java:241) at javax.swing.text.html.parser.Parser.handleText(Parser.java:345) at javax.swing.text.html.parser.Parser.endTag(Parser.java:427) at javax.swing.text.html.parser.Parser.parseTag(Parser.java:1804) at javax.swing.text.html.parser.Parser.parseContent(Parser.java:2044) at javax.swing.text.html.parser.Parser.parse(Parser.java:2211) at javax.swing.text.html.parser.DocumentParser.parse(DocumentParser.java:105) at javax.swing.text.html.parser.ParserDelegator.parse(ParserDelegator.java:73) at com.sun.java.help.search.HTMLIndexerKit.parse(HTMLIndexerKit.java:116) at com.sun.java.help.search.Indexer.parseFile(Indexer.java:259) at com.sun.java.help.search.Indexer.compile(Indexer.java:219) at com.sun.java.help.search.Indexer.main(Indexer.java:87)

Using the source available from subversion, we've localized the issue to a very subtle bug in com.sun.java.help.search.BitSet.concatenate():

162. int tp = _free - 1; // target 163. int sp = 0; // source 164. do

{ 165. _array[tp++] |= bb._array[sp] >>> (NBits - _avail); 166. _array[tp] = bb._array[sp++] << _avail; 167. }

168. while (sp < bb._free);

This can lead to an ArrayIndexOutOfBoundsException when the _free index for the incoming BitBuffer is outside the bounds of the target BitBuffer. We're not sure how the length of the file affects this result, but we did create the following test case to confirm:

(1) Change the initial array size to 3 (instead of 256, to avoid the need for filling up one buffer with 256 words). (2) Define a static main method as follows:

public static void main (String[] args) { / Instantiate both buffers. Note that both buffers can contain up to 96 bits (3 words 32 bits per word) */ BitBuffer target = new BitBuffer(), source = new BitBuffer();

/ Add 8 bits to target – this changes the number of available bits and changes the _free index / target.append(0xAA, 8);

/ Add 72 bits to source / source.append(0x11223344, 32); source.append(0x55667788, 32); source.append(0x99, 8);

/ Note here that target has enough room to consume all bits in source (72 + 8 < 96), so the target buffer will not be resized in concatenate () /

/ Flush outstanding bits in both buffers to array / target.close(); source.close();

/ Invoke concatenate () and take note of the ArrayIndexOutOfBoundsException / target.concatenate(source); }

We believe the issue can be fixed by counting the number of remaining bits instead of watching the word pointer in the do-while loop:

162. int tp = _free - 1; // target 163. int sp = 0; // source 164. int remaining = bb.bitCount(); // <-- Begin with all bits in source remaining 165. do { 166. _array[tp++] |= bb._array[sp] >>> (NBits - _avail); 167. remaining = _avail; // <- Deduct bits added to target from remaining 168. if (remaining > 0) // <-- Are there bits remaining after the move? 169.

{ 170. _array[tp] = bb._array[sp++] << _avail; 171. remaining -= NBits - _avail; // <-- Deduct bits added to target 172. }

173. } 174. while (remaining > 0); // Do this while there are bits remaining

We made this change and rebuilt the JavaHelp project, but making this change seems to have led to yet another exception:

java.lang.ArrayIndexOutOfBoundsException: 17 at com.sun.java.help.search.Decompressor.ascendingDecode(Decompressor.java:176) at com.sun.java.help.search.DocumentLists$MicroIndex.next(DocumentLists.java:105) at com.sun.java.help.search.DocumentLists$MicroIndex.openDocumentIndex(DocumentLists.java:137) at com.sun.java.help.search.DocumentLists$MicroIndex.(DocumentLists.java:79) at com.sun.java.help.search.DocumentLists.(DocumentLists.java:194) at com.sun.java.help.search.DocumentLists.invert(DocumentLists.java:270) at com.sun.java.help.search.DefaultIndexBuilder.close(DefaultIndexBuilder.java:130) at com.sun.java.help.search.Indexer.compile(Indexer.java:242) at com.sun.java.help.search.Indexer.main(Indexer.java:87)

With this exception, the index is not properly built, and so we rolled back our change in BitBuffer.java.

Is there any way this issue can be fixed? Is there any design documentation that we might use to determine how this library works? The software code itself is very scarcely commented, making it very difficult to compare the intended functionality with the actual behavior.

Environment

n/a

Affected Versions

[current]

javaee / javahelp

Bug in com.sun.java.help.search.BitSet.java leads to a long trail of exceptions #38

Environment

Affected Versions