search

attic-labs / noms

The versioned, forkable, syncable database
Apache License 2.0
7.44k stars 266 forks source link

Stop noms sync from destroying locality #3659

Closed cmasone-attic closed 7 years ago

cmasone-attic commented 7 years ago

This patch implements a new strategy for Pull() that pulls the chunks from a given level of the graph over in the order they'll be encountered by clients reading the graph.

Fixes #2968

cmasone-attic commented 7 years ago

PTAL.

The workload I tried this on was using csv-import to import good ol' sfcrime.csv, and then syncing it to a new DB using ToT and the new code. In the former case, the frag tool at https://github.com/attic-labs/noms/pull/3658 reported that one would need almost 18x the optimal number of reads to get the data out. With the new code, the new frag tool reported near-optimal results!

ghost commented 7 years ago

Awesome to see this finally closed, BTW. \o/