Create large dataset for benchmarking

NeowayLabs / neosearch

Full Text Search Library

30 stars 4 forks source link

Create large dataset for benchmarking #3

Open katcipis opened 9 years ago

katcipis commented 9 years ago

We can use:

https://github.com/tiago4orion/DataGen

To generate lots of data and then we can evaluate how fast we really are :-)

i4ki commented 9 years ago

At the moment, the DataGen only generates CSV files, but we store JSON-structured data in neosearch... I've generated 10G of CSV's on my machine, but suffered a lot to convert to JSON (split + converting 200mb at a time).

We need add support for JSON output in DataGen! I'll do that soon!

Thanks!