[BUG] API endpoints failing with illegal_argument_exception

[lpiob]

Describe the bug Ocassionaly, and without an established cause, following endpoints:

• /_stats
• /_cat/indices
• /_cluster/stats

return 400 Bad Request with following error:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -9223372036854028697b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -9223372036854028697b"},"status":400}

Other API methods - especially related to searching documents, or checking overall cluster health - work fine.

The negative value is not constant. Other values that we've encountered: -9223372036854704795b, -9223372036853398293b.

The error goes away after restarting the nodes, and sometimes the error goes away on it's own.

OpenSearch nodes do not log any information when such error is reported.

To Reproduce Reproduction steps - unknown.

Please advise how to debug further if the error returns.

Expected behavior More verbose errors.

Plugins opensearch-alerting opensearch-anomaly-detection opensearch-asynchronous-search opensearch-cross-cluster-replication opensearch-geospatial opensearch-index-management opensearch-job-scheduler opensearch-knn opensearch-ml opensearch-notifications opensearch-notifications-core opensearch-observability opensearch-performance-analyzer opensearch-reports-scheduler opensearch-sql

Host/Environment (please complete the following information):

• OpenSearch 2.3.0
• 3 nodes, each with master,ingest,data,remote_cluster_client roles, 4GB of memory for each of them, java memory limits set to 2GB, 100GB disks with about 1GB used for each node, 9 indices, each with 2 shards and 1 replica, about 2 000 000 documents total.
• Kubernetes installation with https://opensearch-project.github.io/helm-charts/ helm chart

Additional context Test requests and responses recorded in https://gist.github.com/lpiob/56d28fca8378bb94ead9a30893a5dfe8

[bowenlan-amzn]

4474 looks similar

[d3vzer0]

Facing the same issue atm with 5 nodes on a vanilla (ie. non-k8s) deployment. Sometimes the issue just disappears by itself 🤔