mafintosh / hyperdb

Distributed scalable database
MIT License
753 stars 75 forks source link

iterate all keys #17

Closed max-mapper closed 6 years ago

max-mapper commented 6 years ago

i have a couple use cases where i need to basically .createReadStream() all keys in the entire hyperdb, e.g. for exporting the whole thing to a csv

hackergrrl commented 6 years ago

If you just need current keys and not historic values, you could do db.createDiffStream('/') and filter out the delete entries, which will do a diff from HEAD to GENESIS (or whatever we call beginning of time).

hackergrrl commented 6 years ago

If you need all historic values (basically, the whole DAG) you'll be able to use a history stream. (soon)

jayrbolton commented 6 years ago

what do you think about having a method like db.find("*") (or eg. db.find("/*/prefix/*/wat/*"))that accepts splats? It would return a readable stream that gets data for each found node.

if yall think this is a good idea, i would enjoy figuring out the trie traversal necessary to make this work. I was wanting something like this recently myself.

hackergrrl commented 6 years ago

@jayrbolton I think it'd be a cool API to have. :+1:

mafintosh commented 6 years ago

in master

mafintosh commented 6 years ago

@jayrbolton the iterators in master are sorted by hash which means you can actually efficiently implement the /*/prefix/*/... pattern. let me know if you need some pointers and i'll be happy to help