Improve performance when storing/loading from ByteArrays

[BenjaminWarnke]

Additional context I am implementing a database. Therefore my use case is:

1. Instantiate a BigInteger once from a String
2. convert it to ByteArray, save it to a File (once!) ... (right now the BigInteger is garbage-collected here)
3. read it many many (million) times from that ByteArray/File
4. do calculations with it (+-*/><= ...)
5. eventually print it as String

Is your feature request related to a problem? Please describe. In step 3 I am loading the BigInteger from an ByteArray - each time a new java-class is created. This might be ok for a few values, but it is very bad for the garbage collector, if I have millions/billions of these.

Describe the solution you'd like I'd like to skip step 3 entirely. I want to call the calculation functions (+- ...) directly on the ByteArrays (possibly supply offset and length within the byte array, to support storing multiple BigIntegers in a single ByteArray) without instantiating any classes.

Describe alternatives you've considered Reusing an existing BigInteger class. Right now the function "fromByteArray" returns a new class instance. We could add another parameter with an "old" class instance, which we want to override with the contents of the new ByteArray. This would reduce the garbage collector load significantly as well.

The same applies for BigDecimal and the other targets like JS.

[ionspin]

Hi Benjamin,

Thanks for bringing this issue to attention, there was already some work done on improving the allocations (#28), but I am aware that there is much more to be done.

Your proposal to reduce allocations by reusing arrays is definitely a promising way to go, I even think I was experimenting with it at some point, but didn't have time to dig in.

I want to call the calculation functions (+- ...) directly on the ByteArrays (possibly supply offset and length within the byte array, to support storing multiple BigIntegers in a single ByteArray) without instantiating any classes.

One point of inspiration could be Java MutableBigInteger implementation that behaves exactly like you describe, with indexes and array reuse.

Implementing a counterpart in this library would be great, but also a lot of work, starting with defining the interface for the operations, since now they would also take the "old" instance.

I have currently very limited time to work on this, so contributions are welcome! If you want to help out and have time to work on this, please tell me so we can set up a project or something similar, as I think there would be a lot of work and communication on who does what and how would be very important.

Once again, thanks!

[ionspin]

I forgot to mention this:

I want to call the calculation functions (+- ...) directly on the ByteArrays (possibly supply offset and length within the byte array, to support storing multiple BigIntegers in a single ByteArray) without instantiating any classes.

You would incur performance penalty here because hardware is usually optimized for arithmetic operations on words instead of bytes, so you would need at least one conversion from ByteArray to ULongArray which is the backing array in BigNum at the moment.

[BenjaminWarnke]

You would incur performance penalty here because hardware is usually optimized for arithmetic operations on words instead of bytes, so you would need at least one conversion from ByteArray to ULongArray which is the backing array in BigNum at the moment.

In JVM and JS I agree, that we need to convert the data to (U)LongArray, but we could at least reuse those LongArrays there. In Kotlin Native we can cast the ByteArray to a LongArray without actually copy the data around.