The IDTable code does some preprocessing of the note ids and sorts them in ascending integer order. Category note ids are supposed to be added at the end of the IDTable. They have the NOTEID_CATEGORY = 0x80000000L bit set, which unfortunately results in negative integers and the wrong id sorting.
For performance reasons we check the resulting list of sorted note ids for consecutive id ranges and insert them with IDInsertRange and the "addToEnd" argument to true when the IDTable is newly created.
This results in an invalid IDTable structure, because we append higher unsigned integers (the category ids) before lower ones (the document ids).
The basic idea of an IDTable is that it's always in ascending order and we are breaking this expectation, leading to unexpected behavior.
The bug can be reproduced by comparing the result of IDTable.toArray() to IDTable.iterator(). IDTable.toArray() uses the C method IDEnumerate internally which can cope with the wrong order. IDTable.iterator() uses IDScan instead which stops traversing the IDTable too early.
To fix the bug, note id sorting must be changed to treat the note ids as unsigned integers.
The IDTable code does some preprocessing of the note ids and sorts them in ascending integer order. Category note ids are supposed to be added at the end of the IDTable. They have the NOTEID_CATEGORY = 0x80000000L bit set, which unfortunately results in negative integers and the wrong id sorting.
For performance reasons we check the resulting list of sorted note ids for consecutive id ranges and insert them with IDInsertRange and the "addToEnd" argument to true when the IDTable is newly created. This results in an invalid IDTable structure, because we append higher unsigned integers (the category ids) before lower ones (the document ids).
The basic idea of an IDTable is that it's always in ascending order and we are breaking this expectation, leading to unexpected behavior.
The bug can be reproduced by comparing the result of IDTable.toArray() to IDTable.iterator(). IDTable.toArray() uses the C method IDEnumerate internally which can cope with the wrong order. IDTable.iterator() uses IDScan instead which stops traversing the IDTable too early.
To fix the bug, note id sorting must be changed to treat the note ids as unsigned integers.