search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.68k stars 1.79k forks source link

[BUG] ApplyCommitRequest failures when using custom dictionary of size 25MB #5727

Open rahulkarajgikar opened 1 year ago

rahulkarajgikar commented 1 year ago

Describe the bug ApplyCommitRequest is taking too long when the cluster is using custom dictionaries of large size > 25MB[ more specifically, IndicesClusterStateService timing out]

[ApplyCommitRequest{term=10, version=19029, sourceNode={5c557a40cb3ee7836639f99b7c0e4100}{6hKxNzvgRQezm7qLqkeixg}{73gx871cQcirs9V4VOAwAA}{10.212.28.135}{10.212.28.135:9300}{imr}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=true, zone=us-east-1a, cross_cluster_transport_address=2600:1f10:41c2:8f81:81f5:6f66:b680:417c, shard_indexing_pressure_enabled=true, di_number=3}}] took [1.5m] which is above the warn threshold of [30s]:

[running applier [org.opensearch.indices.cluster.IndicesClusterStateService@38d87a65]] took [95196ms],

This is causing data nodes to not process the cluster state in time, so they get dropped out of the cluster.

Expected behavior Data nodes to start dropping out of cluster when custom dictionary size becomes too large (>=25MB)

minalsha commented 1 year ago

@rahulkarajgikar are you trying to load this custom dictionary from a plugin or via OpenSearch?

minalsha commented 1 year ago

@rahulkarajgikar can you please help answer the above question?

rahulkarajgikar commented 1 year ago

Custom dictionary was loaded via opensearch: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/custom-packages.html#custom-packages-using

shwetathareja commented 1 year ago

It looks like IndicesClusterStateService during commit phase could be parsing the complete mappings again (which is 25MB) and is taking longer.