Don't use deep pagination to fetch all entries

enonic / xp

Enonic XP

https://enonic.com

GNU General Public License v3.0

202 stars 34 forks source link

Don't use deep pagination to fetch all entries #7139

Closed GlennRicaud closed 4 years ago

GlennRicaud commented 5 years ago

Use ES 'Search Request Scroll' instead of Deep Pagination for processes like dump and export Linked to https://github.com/enonic/xp/issues/7124

rymsha commented 4 years ago

I rebranded it to a bug. There are certain issues with quickfix.

rymsha commented 4 years ago

"Search Request Scroll" is still not the best approach in cases when data must be temporary stored in memory (for extra processing - like in Vacuum). For huge datasets (like 5M versions in one repository) it either consumes entire memory or risks expiring cursor in elasticsearch.

Next best candidate algorithm is batching with sort by unique value and filtering of already processed entries by range filter.