Keep reporting errors

[ysq5202121]

Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" [11:17:01 FTL] Error processing action: SaveEventAction Message: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]"

System.ApplicationException: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]"

---> Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" --- End of inner exception stack trace --- at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.IndexDocumentsAsync(IReadOnlyCollection1 documents, Boolean isCreateOperation, ICommandOptions options) at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.AddAsync(IEnumerable1 documents, ICommandOptions options) at Exceptionless.Core.Pipeline.SaveEventAction.ProcessBatchAsync(ICollection`1 contexts) in E:\DataCenter\Other\Exceptionless\Exceptionless-7.0.9\src\Exceptionless.Core\Pipeline\040_SaveEventAction.cs:line 23 [11:17:01 ERR] Error processing EventPost 48d13c96b0db4fa9b6f889ae1ca1c6bb q\f0c\f0c0259274ab4da79a8e2ee990b8c5ef.payload: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]"

System.ApplicationException: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]"

---> Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b54?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509818352/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509814760/486.1mb], new bytes reserved: [3592/3.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=3592/3.5kb, accounting=17669482/16.8mb]" --- End of inner exception stack trace --- at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.IndexDocumentsAsync(IReadOnlyCollection1 documents, Boolean isCreateOperation, ICommandOptions options) at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.AddAsync(IEnumerable1 documents, ICommandOptions options) at Exceptionless.Core.Pipeline.SaveEventAction.ProcessBatchAsync(ICollection`1 contexts) in E:\DataCenter\Other\Exceptionless\Exceptionless-7.0.9\src\Exceptionless.Core\Pipeline\040_SaveEventAction.cs:line 23 [11:17:01 INF] Processing post: id=d68812ac14b3449cb6496e63887ddacf path=q\83a\83aac8198608449da5717d003fd5d85a.payload project=6109ef03dc364208705e0ba7 ip=116.6.100.46 v=2 agent=exceptionless/4.6.2.0 [11:17:01 ERR] Error adding document Original: [ElasticsearchClientException] Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" Server Error (Index=null): [parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb] [429] PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false

Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" [11:17:01 FTL] Error processing action: SaveEventAction Message: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]"

System.ApplicationException: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]"

---> Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" --- End of inner exception stack trace --- at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.IndexDocumentsAsync(IReadOnlyCollection1 documents, Boolean isCreateOperation, ICommandOptions options) at Foundatio.Repositories.Elasticsearch.ElasticRepositoryBase1.AddAsync(IEnumerable1 documents, ICommandOptions options) at Exceptionless.Core.Pipeline.SaveEventAction.ProcessBatchAsync(ICollection`1 contexts) in E:\DataCenter\Other\Exceptionless\Exceptionless-7.0.9\src\Exceptionless.Core\Pipeline\040_SaveEventAction.cs:line 23 [11:17:01 ERR] Error processing EventPost d68812ac14b3449cb6496e63887ddacf q\83a\83aac8198608449da5717d003fd5d85a.payload: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]"

System.ApplicationException: Original: (429 - ElasticsearchClientException) Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" Server: (429) Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]"

---> Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 429 from: PUT /prod-events-2021.10.08/_doc/615fb82c63d06a1a20b42b55?op_type=create&pipeline=events-pipeline&refresh=false. ServerError: Type: circuit_breaking_exception Reason: "[parent] Data too large, data for [] would be [509908910/486.2mb], which is larger than the limit of [493030604/470.1mb], real usage: [509906296/486.2mb], new bytes reserved: [2614/2.5kb], usages [request=559576/546.4kb, fielddata=188218/183.8kb, in_flight_requests=2614/2.5kb, accounting=17669482/16.8mb]" --- End of inner exception stack trace --- at

[ysq5202121]

Why are there Chinese and English

[niemyjski]

This seems related to https://github.com/exceptionless/Exceptionless/issues/965, it looks like your Elasticsearch instance doesn't have enough resources as Elasticsearch is applying backpressure and not allowing documents to be indexed. I would try adding additional nodes or CPU/Memory to Elasticsearch .

As for the localization, they should all be in one language, does this always happen?

[ysq5202121]

Es caused by insufficient memory

[ysq5202121]

“[parent] Data too large” I adjusted the ES memory to 8g without this problem

[ysq5202121]

Chinese and English problems, refresh easy to appear in English

[niemyjski]

Thanks for the follow up and glad you got it worked out by bumping the memory.