Sphinx Configuration for Multi server application

Haseeb717 commented 3 years ago

My website using 5 application servers and have a separate sphinx server which ip I have given in thinking_sphinx.yml Issue is that when I create or update any record it give an error from application servers. As it wants to run delta index but i have not rebuild sphinx on application servers so no bin and delta files.

I want that delta index also happens on sphinx server so application servers will perform better. Is that possible or not? If not then what’s the benefit of keeping sphinx server as a standalone server.

pat commented 3 years ago

Hi Haseeb

When you've got multiple application servers and you want up-to-date indices, there are two options:

Use deltas via a background job worker (such as Sidekiq or DelayedJob), and use the corresponding TS gems ts-sidekiq-delta or ts-delayed-delta. Make sure the background job worker is on the same machine as Sphinx, so it processes the deltas there.
Or, switch to real-time indices rather than SQL-backed indices - which means that changes to your data are communicated to Sphinx using its SphinxQL protocol (instead of needing access to the same file system) via ActiveRecord callbacks.

If real-time indices work for your situation, then that's what I would recommend, as it's better-supported by Sphinx (whereas delta indices fail if you're using Sphinx 3.x and PostgreSQL as your database). But if you stick with Sphinx 2.2.11, then either solution should work well enough.

Haseeb717 commented 3 years ago

Hi @pat Thank you for your reply :)

I have two confusion regarding using background job . I am using sidekiq background job and updated my indices files like

ThinkingSphinx::Index.define(:users,
  :with  => :active_record,
  :delta => ThinkingSphinx::Deltas::SidekiqDelta
) do

end

1) After how much time background job will run? 2) How can I make sure that this background job will run on sphinx server as changes are happening on app server and as per thinking sphinx documentation Therefore when the job for the delta index runs, it will only run on the app server that actually processes the job. . Is it possible that we run background job only on sphinx server even changes happens on app servers.

Thank you again :)

pat commented 3 years ago

You'll need to have Sidekiq (and so, a copy of your app) running on your Sphinx server - and that Sidekiq process should be processing jobs from the ts_delta queue. That Sphinx/Sidekiq server needs to be talking to the same Redis database as your app servers - that way, they'll add the Sidekiq delta jobs to that database, and the Sphinx Sidekiq worker will pick those jobs and process them.

Sidekiq is pretty fast at picking up jobs - so it shouldn't be that much slower than the classic delta style, but it depends on how many jobs are enqueued.

Haseeb717 commented 3 years ago

Great thank you so much :) I have done this but how can I stop delta indexing on app servers? My thinking sphinx config is

production:
  address: x.x.x.x
  mysql41: 9412
  mem_limit: 512M
  bin_path : "/algaraj/software/sphinx/bin/"
  charset_table: "0..9, a..z, _, A..Z->a..z, U+621..U+63a, U+640..U+64a, U+66e..U+66f, U+671..U+6d3, U+6d5, U+6e5..U+6e6, U+6ee..U+6ef, U+6fa..U+6fc, U+6ff"
  searchd_log_file: "log/searchd_production.log"
  query_log_file: "log/searchd_production.query.log"
  pid_file: "/algaraj/software/app/shared/tmp/pids/searchd.production.pid"
  indices_location: "/algaraj/software/app/shared/tmp/sphinx/production/"
  configuration_file: "/algaraj/software/app/shared/tmp/production.sphinx.conf"

When I do any record change on app server...sphinx delta starts perform on that server ... as I have sidekiq and redis setup on app servers also. How can I make sure it happens on sphinx server?

pat commented 3 years ago

Is each app server using their own Redis instance? Or are they using a shared Redis instance? Because it'll need to be shared, and then you'll need to make sure that only the Sidekiq instance on the Sphinx server processes the ts_delta queue.

You can run other Sidekiq instances on your app servers if you want, they just can't be allowed to run jobs in the ts_delta queue.

Haseeb717 commented 3 years ago

Its a shared Redis server I have done exactly mentioned in this gem https://github.com/pat/ts-sidekiq-delta How can I make sure that only the Sidekiq instance on the Sphinx server processes the ts_delta queue?

As I gave sphinx server IP address in thinking_sphinx.yml and set indices like this

ThinkingSphinx::Index.define(:users,
  :with  => :active_record,
  :delta => ThinkingSphinx::Deltas::SidekiqDelta
) do

end

It should work or I need to write and sidekiq worker also or any other change?

pat commented 3 years ago

Your Index definition is looking fine - it's the Sidekiq configuration you'll need to check: https://github.com/mperham/sidekiq/wiki/Advanced-Options#queues

So, if you're configuring Sidekiq via command-line arguments, use -q ts_delta on your Sphinx server, and -q default or whatever other queues you're using on your app servers. Or if it's via configuration files, that's fine too, just make sure that your app server Sidekiq configuration file doesn't include ts_delta in its list of queues, and the Sphinx server file does.

pat commented 2 years ago

It's been almost two months, so I'm going to close this issue - but, we can reopen it if there's further questions or thoughts, so do get in touch if help's needed :)

pat / thinking-sphinx

Sphinx Configuration for Multi server application #1198