Fix behaviour of getApproximateSize()

quadstorejs / quadstore

A LevelDB-backed graph database for JS runtimes (Node.js, Deno, browsers, ...) supporting SPARQL queries and the RDF/JS interface.

https://github.com/quadstorejs/quadstore

MIT License

203 stars 14 forks source link

Fix behaviour of getApproximateSize() #54

Closed jacoscaz closed 7 years ago

jacoscaz commented 7 years ago

There are two issues with QuadStore.prototype.getApproximateSize():

[x] LevelDB's .approximateSize() returns the number of bytes, not records. How to convert? I've read somewhere that levelgraph has tried using approximateSize/128 as a reference number. Perhaps we could keep track of the current avg. size of our quads? Would probably need to be re-computed every now and then, though.
[ ] LevelDB's .approximateSize() seems to return 0 for small(-er) datasets.

elf-pavlik commented 7 years ago

I've read somewhere that levelgraph has tried using approximateSize/128 as a reference number.

It looks like @pietercolpaert worked on that https://github.com/levelgraph/levelgraph/pull/72#issuecomment-46307997 maybe he will have some helpful suggestions.

jacoscaz commented 7 years ago

I've done tests to verify the 128 value and it does seem to hold as a good average number, although there's a high variation between different datasets. As for the 0 value for smaller datasets, this is something that needs more research and testing, perhaps with someone from leveldb's team. It is not necessary for v3.0 release, though, as the ldf endpoint does count returned quads and corrects the estimated count accordingly.