meilisearch / meilisearch-laravel-scout

MeiliSearch integration for Laravel Scout
https://github.com/meilisearch/MeiliSearch
MIT License
465 stars 43 forks source link

Update of existing records #124

Closed hope22 closed 3 years ago

hope22 commented 3 years ago

I have meilisearch working via Laravel Forge on its own dedicated ec2 instance.

Everything works great.

Apart from when a record is updated via ->save() eloquent method.

The record on the meilisearch server doesn't change.

I'm updating via amazon sqs queues.

I've been checking this via Postman by doing a direct get request to the server and the records just don't update.

I've hit a dead end in my debugging and wondering if anyone else has a similar issue or there is just a big delay when updating records.

Update

It seems to be the tasks taking up to 5 hours to process a document update. Not sure why this is taking so long or if there is a way I can assign more processes to this process?

{
    "status": "processed",
    "updateId": 11022,
    "type": {
        "name": "DocumentsAddition",
        "number": 1
    },
    "duration": 47.64718841,
    "enqueuedAt": "2021-04-17T11:10:20.522991788Z",
    "processedAt": "2021-04-18T17:43:41.904437172Z"
}
hope22 commented 3 years ago

They seem to be stuck at this status, a few updating every few minutes with a huge backlog.

{
    "status": "enqueued",
    "updateId": 11100,
    "type": {
        "name": "DocumentsAddition",
        "number": 1
    },
    "enqueuedAt": "2021-04-17T12:28:48.694663760Z"
}
curquiza commented 3 years ago

Hello @hope22! I see you have more than 10.000 updates, do you send them at the same time? Is one update contain only one document? You should them several documents at the same time, by batches.

hope22 commented 3 years ago

Hey @curquiza.

I have since started batching updates in groups of 2000 which gives me faster results. My issue seemed to be when single documents were updated they were getting backlogged.

Would be great if there was an API route to clear the black-logged updated then I can just run a fresh import again.

Apart from that, amazing product! Loving it. Well done :)

Can make as resolved.

curquiza commented 3 years ago

Hello @hope22! We are thinking about adding this possibility into MeiliSearch. It will maybe be added some days :)

Thanks for your feedback! Thanks for using MeiliSearch ❤️