search

meilisearch / meilisearch-go

Golang wrapper for the Meilisearch API
https://www.meilisearch.com
MIT License
480 stars 81 forks source link

[Feature] Reduce tasks/data.mdb file size #496

Closed jackbit closed 8 months ago

jackbit commented 8 months ago

Description I have background job that always run every 5 minutes and the data are always changed, so i need to update hundred thousands documents every 5 minutes in meilisearch. So far it's blazing fast but the size of data.ms/tasks/data.mdb keeps getting bigger than the folder data.ms/indexes its self. I set a cron job that delete tasks for any status, it seems no change in tasks/data.mdb file size. I use cronjob curl to delete task:

curl \   
  -X DELETE 'http://localhost:7700/tasks?statuses=*' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer SecretAuthKey'

Basic example I'd love to propose that Meilisearch should reduce data.ms/tasks/data.mdb , if delete tasks api is done, or set configuration to not store tasks after executed, either succeeded, failed, and canceled, example meilisearch --not-store-tasks

curquiza commented 8 months ago

Hello @jackbit

Your feature is not related to this library but to meilisearch itself.

The engine team is currently working on reducing the size of the DB. This is something we work on for v1.5.0 and v1.6.0. This is not our main focus but part of a bigger focus related to indexing performance improvement.

If you have more questions you can join our Discord

I close this issue since it's not related to this library