Integer Overflow in WriteBuffer.java at WriteBuffer.grow() in MVStore 1.4.186

[GoogleCodeExporter]

When used with large commits WriteBuffer.grow() can fail due to an integer 
overflow. The variable 'newCapacity' gets negative and 
'ByteBuffer.allocate(newCapacity)' fails.

I've attached a demo which reassembles the internal state I've seen in my 
debugger. 

Do you need more information? 
Regards, Andreas

Original issue reported on code.google.com by oslothef...@googlemail.com on 25 Mar 2015 at 3:46

Attachments:

• WriteBufferGrowIntOverflow.java

[GoogleCodeExporter]

Thanks a lot for reporting! I have fixed it in trunk now.

What is not clear to me is how did you run into this situation. The write 
buffer would need to be extremely large. Could you describe what you did 
exactly?

Original comment by thomas.t...@gmail.com on 26 Mar 2015 at 7:00

• Changed state: Accepted

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Wow, that was fast - thanks and kudos for that!

For what we where doing:
We're using MvStore directly to store a huge amount of data in three maps. One 
map containing the main data, two others containing indices. 

The values of the main data Map<KEY,Serializable> should be the largest atomic 
piece of data - but on the average stay below 64K. The KEY is byte[16].

The indices are expected to have rather small values on the average but may 
also grow sometimes to larger values.  Indices are Map<KEY,long[]>, Map<long, 
KEY[]>.

Expected sizes are ~12 million KEYs and 1,5 million longs - just to get an idea.

Overall it works pretty well with our use case. The overflow occurred when we 
tried to do a initial full build of the store. Normal updates are then expected 
to be rather small. 

From what I've seen the cause might be that we wanted to use a single commit() 
only at the end of the complete init. As workaround/test we chunked the input 
data and committed at several selected points manually, which made the problem 
to disappear. Using autoCommit() is not an option for us, but might have 
avoided the problem as well.

(corrected version)

Original comment by oslothef...@googlemail.com on 26 Mar 2015 at 9:51

[GoogleCodeExporter]

Should be fixed with version 1.4.187

Original comment by thomas.t...@gmail.com on 10 Apr 2015 at 6:55

• Changed state: Fixed