manticoresoftware / manticoresearch-backup

Repository for Manticore Search backup scripts
GNU General Public License v3.0
3 stars 1 forks source link

Improve decompression logic on restoring compressed backup #76

Closed donhardman closed 10 months ago

donhardman commented 1 year ago

At present, our backup restoration process involves reading the file into memory and decompressing it within the same process. This method can lead to "memory out of limit" errors when dealing with large files. To rectify this, we should adopt a more efficient approach for decompression by leveraging the streams of the zstd extension. This will help minimize memory consumption during the restoration of substantial files.

donhardman commented 10 months ago

Implemented in this pull: https://github.com/manticoresoftware/manticoresearch-backup/pull/85