khonsulabs / bonsaidb

A developer-friendly document database that grows with you, written in Rust
https://bonsaidb.io/
Apache License 2.0
1.01k stars 37 forks source link

Test Large Datasets/Create Open-Library Data Importer #213

Closed ecton closed 2 years ago

ecton commented 2 years ago

This is a new example that will turn into a few things, I think. Initially, it's a way for me to test a large amount of data. The README shows the data sizes being tested. It served as a way for me to feel confident in its behavior with a large, organic dataset.

To adequately do that, more than just counting the records needs to happen. Two uses come to mind:

Of note, running this multiple times causes the ratings set to be appended to each time. This is because the dataset doesn't have a primary key for each row, most likely due to it being anonymized.