API scalability: Analyze Elasticsearch configuration & make recommendation

[lizkrznarich]

Service/repository new-deployment

Describe the current state/issue Currently, ROR has 1 Elastic search cluster with 1 node for each env (dev, staging, prod). There is no redundancy in prod, and CPU hits 100% about once per day.

Describe the desired state/solution Research and make a recommendation for changes to Elasticsearch configuration to add redundancy and scale resourcing horizontally. Related to roadmap item https://github.com/ror-community/ror-roadmap/issues/124

Additional notes

• ROR's indexing rate is very low, but search rate is high
• Search rate and CPU usage are somewhat predictable, but there are large spikes and a general increase as new integrators start using ROR APIs

[lizkrznarich]

Based on the resources below, recommendations include:

• Increase node count in prod cluster to 2
• Set primary and replica shards to 1, so that each node has a replica of the primary shard on the other node
• Increase filesystem cache memory
• Consider adding names and ids fields to main doc via copy-to, rather nested doc
• Consider 1 non-data node for HTTP connections

https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html https://docs.aws.amazon.com/opensearch-service/latest/developerguide/bp.html