search

techfort / LokiJS

javascript embeddable / in-memory database
http:/techfort.github.io/LokiJS
MIT License
6.74k stars 482 forks source link

[Question] Subscribe: performance and efficiency #913

Open sc00 opened 2 years ago

sc00 commented 2 years ago

Hi,

would you recommend using the subscribe feature over updating the UI directly in the frontend in terms of performance and/or efficiency? What do you consider best practice?

I'm currently redoing my SvelteKit/Tauri app with Electron. Observables were no option with my sqlite setup at the time. Now I consider using the subscribe method instead of updating the UI from the frontend. I think this would only be worthwhile if it would lead to better results or had any other advantages that I might be missing :)

Thanks and cheers

Losses commented 2 years ago

If you are talking about the insert / update / error event, the cost of listening to these events are pretty low so it's safe to do that, but if your data updating logic on the Svelte side is heavy, remember throttle the update callback.

The main bottleneck is probably the adapter, if you are selecting an incorrect adapter for a looooooooooot of data, the performance of serialization would be pretty low.

Most of the adapter are built for Node.js so I assume you:

  1. Do not need to persistence the data into the hard disk.
  2. Implemented your own persistence adapter with Rust since you are using Tauri.
  3. Sync your data with the Changes API.

A detailed description would help a lot to let me know what you are expecting from this library.

sc00 commented 2 years ago

If you are talking about the insert / update / error event, the cost of listening to these events are pretty low so it's safe to do that, but if your data updating logic on the Svelte side is heavy, remember throttle the update callback.

The main bottleneck is probably the adapter, if you are selecting an incorrect adapter for a looooooooooot of data, the performance of serialization would be pretty low.

I understand, thanks!

Most of the adapter are built for Node.js so I assume you:

  1. Do not need to persistence the data into the hard disk.
  2. Implemented your own persistence adapter with Rust since you are using Tauri.
  3. Sync your data with the Changes API.

I'm currently moving away from Tauri due to an unrelated issue (I've been using sqlite with the old setup). Now I'm switching over to Electron, so Node it is and LokiJS came into play :) I do need to persist data into the hard disc though!

A detailed description would help a lot to let me know what you are expecting from this library.

I need to store documents in 4 collections. I expect them to hold no more than 5000 documents combined (probably much less on average - maybe around 1000). The documents hold objects with 5-30 properties (1 level of nesting max).

Within the app there's a list of all items for every collection (separate routes). I need to experiment with performance, but I'll probably put a limit on the query. I need to be able to filter each list with a search term that will be compared against a couple of properties.

As of now when I run an operation like insert/update/remove, I don't re-run the query but update the list from the frontend accordingly. Of course this was less convenient to write, but since I now have the option to use an observable, I wonder if I should get rid of all of this code :)

Losses commented 2 years ago

I think your use case is safe, good luck.

BTW, avoid using loki-fs-structured-adapter, it is buggy.