search

dscarpetti / codax

An idiomatic transactional embedded database for clojure
Eclipse Public License 1.0
179 stars 9 forks source link

Return only keys? #24

Closed Frozenlock closed 1 year ago

Frozenlock commented 5 years ago

I have multiple projects for which I only want to know their ID (keys).

Usually I would simply do this (or the Codax equivalent):

(keys projects-db)

Once I have the keys I could use c/get-at to only fetch the information I want.

However this is taking way too much time, most probably because Codax is loading the entire map even when I only need the keys. Is there an alternative approach to obtain the same info without loading everything?

dscarpetti commented 5 years ago

Sorry for the delay. The way that the data storage is structured makes such a key retrieval difficult. Each leaf is essentially stored (and sorted) by a bit representation of its full keypath, which means that you must iterate through all children before you come across the next key in a given map. So, summarily, you are exactly right, Codax is loading the entire map.

One way to accomplish it is by storing a bucket of keys for each data path and updating it in parallel. I am curious if you have attempted something similar in your application code? I think it is a somewhat easier problem to address on an application specific level. The downsides are that it would add overhead on write operations and would probably break backwards compatibility with existing databases.

Another way may be to iterate through the keys without decoding and loading the values in to memory. This would be fairly easy to implement, but I expect would probably still be too slow for your use case. Though it might be worth a shot.