In my tests, the indexing in batches is almost 2x as fast with these settings. Unfortunately, none of these settings seems to affect the query speed very much. Next I’ll being to profile a single query on one of these datasets to see where time is being spent. I’m wondering if some way we lay out data means that we don’t end up getting much benefit from the bloom filter, because we end up needing something from each file anyway. Still investigating…
You may want to try these settings on intrapop [and other cbft apps] and see if they help.
Oh, unfortunately unlike ForestDB, with LevelDB the configuration is at the database level. Above, the 2 caches are 512MB each. So, if your number of shards * 1GB is greater than the available ram, you should probably reduce them.
Some results on the bleve project and leveldb from Mary Schoch...
I played around with some different leveldb configs with bleve-bench. It looks like the following config works best:
{ "write_buffer_size": 536870912, "lru_cache_capacity": 536870912, "bloom_filter_bits_per_key": 10 }
In my tests, the indexing in batches is almost 2x as fast with these settings. Unfortunately, none of these settings seems to affect the query speed very much. Next I’ll being to profile a single query on one of these datasets to see where time is being spent. I’m wondering if some way we lay out data means that we don’t end up getting much benefit from the bloom filter, because we end up needing something from each file anyway. Still investigating…
You may want to try these settings on intrapop [and other cbft apps] and see if they help.
Oh, unfortunately unlike ForestDB, with LevelDB the configuration is at the database level. Above, the 2 caches are 512MB each. So, if your number of shards * 1GB is greater than the available ram, you should probably reduce them.