Performance test of YdbSpliterator

ydb-platform / yoj-project

YDB ORM for Java (YOJ) is a lightweight ORM for immutable entities. It has native support for YDB and is battle-tested.

Apache License 2.0

13 stars 12 forks source link

Performance test of YdbSpliterator #42

Open lavrukov opened 9 months ago

lavrukov commented 9 months ago

YdbSpliterator uses ArrayBlockingQueue<>(1) which produce overhead on threads synchronisation. This decision was made based on the desire to minimize memory usage, but it could be too slow.

We have to know numbers. We need to make a performance test on 1kk amount of data with:

current spliterator
spliterator with ArrayBlockingQueue<>(N) where N in [100, 1000, 10000]
spliterator with non-blocking queue or something which could work better by time

nvamelichev commented 9 months ago

If current ArrayBlockingQueue(1) implementation turns out to be too slow in many practical applications, then the buffer size could be made adjustable in ReadTableParams :-)