opensearch-project / documentation-website

The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.
https://opensearch.org/docs
Apache License 2.0
74 stars 488 forks source link

Concurrent merges section of Performance Tuning could be more helpful #6549

Open gregschohn opened 8 months ago

gregschohn commented 8 months ago

What do you want to do?

I'd like to see the documentation be more precise to prevent users from making incomplete changes or creating contradictory settings values.

Tell us about your request. Provide a summary of the request and all versions that are affected.

I've seen a case where a settings are changed via

PUT /index_name/_settings
{
"index.merge.scheduler.max_merge_count" :  "1",
“index.translog.flush_threshold_size”: “1024MB”
}

results in an exception java.lang.IllegalArgumentException: maxThreadCount (= 4) should be <= maxMergeCount (= 1). However, the documentation for concurrent-merges states that "Merges run in separate threads, and when the maximum number of threads is reached, further merges will wait until a merge thread becomes available."

If that's the case, I don't see why maxThreadCount and maxMergeCount would be related. This section could be more clear to indicate how settings affect each other

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

There's another ticket here that allowed settings to be entered without being fully validated. The exact failure behavior of these invalid settings therefore may be manifested differently.

hdhalter commented 8 months ago

@praveensameneni - Do you know who might be able to help with clarifying this in the documentation?

hdhalter commented 8 months ago

Hi @andrross , @sohami, can you please comment or help clarify this in the documentation? Thanks!

andrross commented 8 months ago

maxThreadCount (= 4) should be <= maxMergeCount (= 1)

@gregschohn I think this means that if your max thread count was larger than max merge count then you would always have idle threads because max merge count would prevent them from being used. How do you suggest we clarify this?