search

opensearch-project / dashboards-maps

:globe_with_meridians: User interface for the backend geospatial plugin which can be integrated with OpenSearch Dashboards and a home for all other map related features.
https://opensearch.org/docs/latest/dashboards/geojson-regionmaps/#custom-vector-maps-with-geojson
Apache License 2.0
17 stars 44 forks source link

[BUG] Dashboards-maps: documents size limit #587

Open cbeaujoin-stellar opened 6 months ago

cbeaujoin-stellar commented 6 months ago

What is the bug?

As described here: https://github.com/opensearch-project/dashboards-maps/pull/141

Update default query document size limit to 1000, the max is from index.max_result_window size,

But in fact limitation is set to 10000 and is not aligned with index.max_result_window:

https://github.com/opensearch-project/dashboards-maps/blob/811e8b096b77c57fb820516262d5c3e3d4f43c3b/public/components/layer_config/documents_config/document_layer_source.tsx#L212

How can one reproduce the bug?

Create a map based on an index with number of documents and index.max_result_window greather than 10000. You can't set a Number of Document greater than 10k.

What is the expected behavior?

UI should be aligned with the spec:

Idealy, maybe the maps plugin should use the scroll API when index.max_result_window > 10k

What is your host/environment?

docker, v2.11.1

Do you have any screenshots?

image

Do you have any additional context?

https://forum.opensearch.org/t/dashboards-maps-documents-size-limit/17983

junqiu-lei commented 6 months ago

@cbeaujoin-stellar Thanks reporting, DOCUMENTS_DEFAULT_REQUEST_NUMBER was by designed to set to 1000 by default so avoid large number of data render issue.

@BionIT When we support multi data source, do you think if it's possible to read the value from index.max_result_window?

cbeaujoin-stellar commented 4 months ago

Hi @junqiu-lei, is this planned for 2.14.0 (May 14th, 2024) ?

junqiu-lei commented 4 months ago

Assign to @vamshin to decide the priority.

vamshin commented 4 months ago

@cbeaujoin-stellar we are looking to have this delivered for 2.15.

cbeaujoin-stellar commented 2 months ago

Do not forget that for next release, it's still very boring to manualy export edit import Maps objects Json file.

junqiu-lei commented 2 months ago

@cbeaujoin-stellar After we looked into this issue, rendering issues can occur when large number of documents are visualized same time, which is why we have set the default document size to 1000. Additionally, the Maps app uses index patterns instead of single indices for data layers. An index pattern can encompass multiple indices, making it impractical to apply a single index's max_result_window setting in this context.

cbeaujoin-stellar commented 2 months ago

I'm running it with 4 indices with 500k documents each, it's working fine.

cbeaujoin-stellar commented 2 months ago

It should at least be configurable according to management/opensearch-dashboards/settings > Custom vector map size