search

AntonShuvaev / elasticsearch4idea

Elasticsearch Plugin for JetBrains IDEs
https://plugins.jetbrains.com/plugin/14512-elasticsearch
59 stars 8 forks source link

Cannot work with large results #49

Closed ostrolucky closed 2 years ago

ostrolucky commented 2 years ago

I wanted to use this plugin to export large index into CSV, having more than 1 billion results. However, when I increase page size beyond 10000, ES complains that result window is too large.

2021-10-26_17-37-03 (1)

Plugin should paginate these results and load data in batch, not attempt to request everything at once.

AntonShuvaev commented 2 years ago

Hi! I will consider adding paginating with search_after to export as Elasticsearch doesn't allow search more than index.max_result_window (10000 by default).

AntonShuvaev commented 2 years ago

I have added the option to export all documents with Scroll API.

ostrolucky commented 2 years ago

Thanks! I was thinking this would be integrated into core transparently, but this way it also seems working. Good to see you respond to issues in timely manner, it gives me affirmation that it was good decision to pay for the licenses!

I see there is still one related problem to this, but I'll create separate issue for it :)