search

pubkey / rxdb

A fast, local first, reactive Database for JavaScript Applications https://rxdb.info/
https://rxdb.info/
Apache License 2.0
21.52k stars 1.06k forks source link

Support bulkUpsert? #2501

Closed jackple closed 4 years ago

jackple commented 4 years ago

I use rxdb for Electron with leveldown I can not find bulkUpsert in the doc. This is the only way I can bulkUpsert:

for (const item of docsData) {
    await this.collection.atomicUpsert(item)
}

It takes a long time! Is there any better idea?

pubkey commented 4 years ago

There is no function for that at the moment. You can make a PR if you have the time.

Btw have you tried running the upserts in parallel? This might be slightly faster.

await Promise.all(
    items.map(
        item => this.collection.atomicUpsert(item)
    )
)

Otherwise I recommend to split it into upserts and inserts be getting the documents first and then you can do a bulk-insert.

jackple commented 4 years ago

@pubkey Thanks

gustavomick commented 4 years ago

@pubkey hi! considering pouch allows bulk, wont bepossible to have a bulk-update for those stores (like pouch) that allow this?, i guess my real question is if is a technical limitation you are already aware and we can not avoid at this point, or just not done yet due to low prio, it seems very important for realtime/offline browser capabilities. btw we really appreciate your work, thanx!

pubkey commented 4 years ago

@gustavomick no there is no technical limitation. But implementing this is not as easy as it looks like. To do writes on existing documents, you must first get the revisions of the old ones. But between getting these revisions and your writes, there could have happened other writes which changed the revisions, so you must also handle occurring conflict errors. I just do not need this for my self atm so I haven't implemented it. If you make a PR, ensure you cover theses edge cases in tests by doing many writes while using the bulkUpsert.

gustavomick commented 4 years ago

y make sense. tbh I don't have time myself either, but wanted to know your insights about this ... as I know I gonna have a blocker some point in the future due already happened to me using pouch directly, but that time I was able to replace it with SQLite (mobile) which increased perf 10x or more. thanks for the feedback if I have to hit this myself I will look for your advice.