search

elastic / hey-apm

Basic load generation for apm-server built on hey
Apache License 2.0
16 stars 16 forks source link

[error] Too many scroll contexts #194

Open cachedout opened 3 years ago

cachedout commented 3 years ago

The Hey apm test benchmark pipeline job is constantly failing with errors like the following:

500 Internal Server Error: [500 Internal Server Error] {"error":{"root_cause":[{"type":"exception","reason":"Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting."}

cc: @axw

axw commented 3 years ago

Looking at the log, I think this is probably related to the delete_by_query request used to delete old data before each benchmark run: https://github.com/elastic/hey-apm/blob/4ac0341c0008086085c21dc1ac0c6958699a3714/es/api.go#L129. Going to have to come back to this one, I don't think it's straightforward to fix.

axw commented 3 years ago

I manually deleted a bunch of stale indices from the cluster, should get us unstuck for now. It'll probably happen again, until we stop sending the apm-server workload events to the same cluster (https://github.com/elastic/hey-apm/issues/167).