I faced issue that if my initial YDoc starts from 3-5Mb this can lead to serious memory usage by IndexedDB on each update.
First steps
I investigated a lot of discussions, issues, forks etc. Understood each situation in detail.
What a problem?
People wait that this just should works good. But everyone has their own expectations. I think each tool need more detailed documentation about such processes.
After bunch hours of learning code and other situations I realized that inside y-indexeddb exists logic for optimization memory usage. It triggers under certain condition. You just need to reach it. But not so simple... 🫤
PREFERRED_TRIM_SIZE
On each document update this constant used for checking whether IndexedDB exceed to this limit. If yes - do trim.
Trimming - reduce data usage of IndexedDB.
PREFERRED_TRIM_SIZE equals 500
_dbsize equals records count in IndexedDB.
By simple words:
if (idbPersistence._dbsize >= PREFERRED_TRIM_SIZE) {
// do trim
}
So, for triggering data trimming I need to do 500 YDoc updates to update IndexedDb 500 times respectively.
Okay, whats next?
If your update size is 12b it really doesn't matter - 12b 500 = 6000b - 6kb
But if your update is 4Mb here is interesting - 4Mb 500 = 2000Mb - 2Gb 🥲
And yeah, this is expected behavior.
So what do you want man? 🤔
I think the best approach would be to make PREFERRED_TRIM_SIZE configurable. For example decrease it value from 500 to 100. (4Mb*100 = 400Mb)
Main idea that each can configure PREFERRED_TRIM_SIZE by their data size.
If you know that your updates big - you decrease PREFERRED_TRIM_SIZE for triggering trimming earlier.
So this PR:
Makes PREFERRED_TRIM_SIZE configurable from IndexeddbPersistence instance via passing options object
Example:
new IndexeddbPersistence("my-document-id", doc, {
PREFERRED_TRIM_SIZE: 100
})
@dmonad, I would be glad if you can provide feedback for my contribution!
Why
I faced issue that if my initial
YDoc
starts from 3-5Mb this can lead to serious memory usage by IndexedDB on each update.First steps
I investigated a lot of discussions, issues, forks etc. Understood each situation in detail.
What a problem?
People wait that this just should works good. But everyone has their own expectations. I think each tool need more detailed documentation about such processes.
After bunch hours of learning code and other situations I realized that inside
y-indexeddb
exists logic for optimization memory usage. It triggers under certain condition. You just need to reach it. But not so simple... 🫤PREFERRED_TRIM_SIZE
On each document update this constant used for checking whether IndexedDB exceed to this limit. If yes - do trim. Trimming - reduce data usage of IndexedDB.
PREFERRED_TRIM_SIZE
equals500
_dbsize
equals records count in IndexedDB.By simple words:
So, for triggering data trimming I need to do
500
YDoc updates to update IndexedDb500
times respectively.Okay, whats next?
If your update size is 12b it really doesn't matter - 12b 500 = 6000b - 6kb But if your update is 4Mb here is interesting - 4Mb 500 = 2000Mb - 2Gb 🥲
And yeah, this is expected behavior.
So what do you want man? 🤔
I think the best approach would be to make
PREFERRED_TRIM_SIZE
configurable. For example decrease it value from500
to100
. (4Mb*100 = 400Mb) Main idea that each can configurePREFERRED_TRIM_SIZE
by their data size. If you know that your updates big - you decreasePREFERRED_TRIM_SIZE
for triggering trimming earlier.So this PR:
PREFERRED_TRIM_SIZE
configurable fromIndexeddbPersistence
instance via passing options objectExample:
@dmonad, I would be glad if you can provide feedback for my contribution!
Peace ✌️