search

blackstork-io / fabric

An open-source command-line tool for reporting workflow automation and a configuration language for reusable templates. Reporting-as-Code
https://blackstork.io/fabric/
Apache License 2.0
10 stars 0 forks source link

Page through the results in `data.elasticsearch` data source #167

Closed traut closed 4 days ago

traut commented 3 weeks ago

Description

Elasticsearch caps the result window for a search request at 10K results. If the size is set to more than 10K, the error is returned:

[400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way
to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards
failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":".internal.alerts-security.alerts-default-000001","node":"YEi3-SiNQ4-2ZM1tVOYKAg","reason":{"type":"illegal_argument_exception","reason":"Result window is too large, from
+ size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level
setting."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the scroll api for a more efficient way to request large data sets. This
limit can be set by changing the [index.max_result_window] index level setting.","caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [50000]. See the
scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}},"status":400}

Environment

Fabric version: 0.4.1 elastic plugin version: 0.4.1

Steps to Reproduce

The error can be reproduced by settings size value to a number higher than 10000.

Expected Behavior

Fabric should support any non-negative int value in size attribute, with the limiting factor being a gRPC call message size (20mb).

Additional Context