planetfederal / registry

pycsw + elasticsearch + opensearch + swagger - Based on HHyperMap
MIT License
11 stars 9 forks source link

Measure registry's performance on 10K, 50K, 100K, 200K, 500K and 1M records #119

Closed ingenieroariel closed 7 years ago

ingenieroariel commented 7 years ago

Create configuration recommendations in the official docs based on the number of layers, how many shards, what kind of server, etc.

ingenieroariel commented 7 years ago

For this task we downloaded around a million from Data.gov.

The numbers obtained were these:

  1. 50K layers can be loaded under a minute in a server with 4GB of RAM.
  2. Many elasticsearch nodes can be put together in the same server. But in aggregate, they should not use more than half the total ram.
  3. Performance scales almost linearly with the number of nodes as long as 100K per node is not exceeded.
screen shot 2017-04-19 at 10 31 48 am