meilisearch / meilisearch-rails

Meilisearch integration for Ruby on Rails
https://www.meilisearch.com
MIT License
308 stars 47 forks source link

How does Meilisearch Rails detect changes #258

Closed doutatsu closed 1 year ago

doutatsu commented 1 year ago

I am trying to understand how the gem is aware of changes made to the indexable model. It seems to me that it just relies on Rails hooks, rather than looking at the database changes themselves. The reason being, is that I've noticed that a direct SQL made from a different model on the indexable model doesn't trigger a refresh of the index. I also do a lot of direct queries for performance reasons, so while I didn't check for all, I presume they don't trigger the changes either.

Hence my question - and if it indeed doesn't work directly with the database, what would be the advised suggestion to working with direct queries, that update models that need indexing?

brunoocasali commented 1 year ago

Hi @doutatsu yes, this gem relies exclusively on the database hooks.

I assume you don't have the AR object in your hands since you do the queries directly. Because you could run object.index!.

Another idea is to call Model.where('updated_at > ?', 10.minutes.ago).reindex! and you can call that in background worker for example. Did you try this using sidekiq for it? https://github.com/meilisearch/meilisearch-rails#queues--background-jobs (still relies on the hooks to trigger the job, but the processing is done in the background)

But in fact, there is no easy way to accomplish that if you don't trigger the hooks :/

That being said, you should rely on external processing to check the database. I would only recommend doing that if you were outside Rails, because the Rails gem has a lot of benefits and does a lot of stuff for you. But if you have a particular use case, you can try this tool https://github.com/meilisync/meilisync which is a tool created by a contributor where you can configure a watcher process so it can handle the indexing for you. I need to say that we don't maintain it yet, but we are going to invest more time in it in the future.

The only problem I see about you using it is the ETL process you must do if you want to keep using meilisearch-rails because I'm not sure if the meilisync will index your table data in the same way the meilisearch-rails does so that you can have some conflicts there.

Let me know if that helps. ;)

doutatsu commented 1 year ago

Playing around, it does seem like I just have to manually call index! on specific records I know would have an update, after I run a particular query. It is a bit annoying making sure to re-run index when appropriate, but so far it seems to be working alright. I also already use sidekiq to process these jobs in the background. Thanks for giving more context though!