ActiveRecord default_scope ignored

[pioz]

I've a Request class:

class Request < ActiveRecord::Base
    default_scope -> { where('user_id IS NOT NULL') }
    ...
end

When I search with Thinking Sphinx 3.1.0 Request.search I get all requests, ignoring the default scope.

Is there a way to consider the default scope?

[pat]

You'd need to implement this with Sphinx scopes as well - standard scopes do not apply for searches. Something like the following should do the trick

# in the index definition, if you don't already have it:
has user_id

# in the Request model:
include ThinkingSphinx::Scopes

sphinx_scope(:search_with_users) { {:without => {:user_id => 0}} }
default_sphinx_scope :search_with_users

[pioz]

Ok, thanks for reply :D. But it would be nice an option to use active record default scopes.

[pioz]

I've try to apply your solution but seems dont works... I need to exclude all records with user_id setted to NULL and not to 0. But if I use without: {user_id: nil} I get ThinkingSphinx::SyntaxError

[pioz]

Wait, I've read that Sphinx treats NULLs as 0's. So my problem is this: In my class User I've a hook:

class User
  has_many :requests
  before_destroy do
    self.requests.each do |r|
      r.update_attributes(user_id: nil)
    end
  end
end

Now if I destroy an user and then I use Request.search (with default_sphinx_scope {without: {user_id: 0}}) I get all requests. But If I run rake ts:rebuild, I get only the requests without user_id = 0.

How I can the right requests without run rake ts:rebuild?

[pat]

If you want changes to be reflected immediately (or very close to it), you'll need to use either deltas or real-time indices.

[pat]

As for AcitveRecord default scopes - it's not an option, as there's no way an SQL statement can be guaranteed to work as a SphinxQL statement (different schemas, different syntax). It also can't be applied when translating Sphinx results into ActiveRecord objects - the underlying pagination would be incorrect if some search results are removed due to a model's default scope.

[pioz]

Oh, yes, you're right for default_scopes.

Can you give me a link that explain real-time indices?

[pat]

Here's a blog post I wrote on the topic: http://freelancing-gods.com/posts/rewriting_thinking_sphinx_introducing_realtime_indices

[pioz]

Ok, thanks for the link. But I get some errors when I try to convert the index from :active_record to :real_time Here my index:

ThinkingSphinx::Index.define :post, with: :active_record, delta: true, star: true do
  indexes user.first_name, sortable: :insensitive
  indexes user.last_name, sortable: :insensitive
  indexes talent.name, as: :talent_name
  indexes message

  has 'RADIANS(users.latitude)',  as: :latitude,  type: :float
  has 'RADIANS(users.longitude)', as: :longitude, type: :float
  has user.trustmeter, type: :integer
  has created_at, type: :timestamp
  has talent_id, type: :integer
  has user_id, type: :integer

  set_property min_prefix_len: 3
end

If I change the index with ThinkingSphinx::Index.define :post, with: :real_time, star: true do I get the error unknown local index 'post_core' in search request:

thinking-sphinx (3.1.0) lib/thinking_sphinx/connection.rb:90:in `rescue in query'
thinking-sphinx (3.1.0) lib/thinking_sphinx/connection.rb:93:in `query'
thinking-sphinx (3.1.0) lib/thinking_sphinx/connection.rb:75:in `query_all'
thinking-sphinx (3.1.0) lib/thinking_sphinx/search/batch_inquirer.rb:17:in `block in results'
thinking-sphinx (3.1.0) lib/thinking_sphinx/connection.rb:37:in `block in take'
innertube (1.1.0) lib/innertube.rb:138:in `take'
thinking-sphinx (3.1.0) lib/thinking_sphinx/connection.rb:35:in `take'
thinking-sphinx (3.1.0) lib/thinking_sphinx/search/batch_inquirer.rb:16:in `results'
thinking-sphinx (3.1.0) lib/thinking_sphinx/middlewares/inquirer.rb:9:in `block in call'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.0.3) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `instrument'
thinking-sphinx (3.1.0) lib/thinking_sphinx/search/context.rb:22:in `log'
thinking-sphinx (3.1.0) lib/thinking_sphinx/middlewares/inquirer.rb:8:in `call'
thinking-sphinx (3.1.0) lib/thinking_sphinx/middlewares/geographer.rb:11:in `call'
thinking-sphinx (3.1.0) lib/thinking_sphinx/middlewares/sphinxql.rb:14:in `call'
thinking-sphinx (3.1.0) lib/thinking_sphinx/middlewares/stale_id_filter.rb:10:in `call'
middleware (0.1.0) lib/middleware/runner.rb:31:in `call'
middleware (0.1.0) lib/middleware/builder.rb:102:in `call'
thinking-sphinx (3.1.0) lib/thinking_sphinx/search.rb:67:in `populate'
thinking-sphinx (3.1.0) lib/thinking_sphinx/search.rb:91:in `to_a'
app/views/requests/search.html.haml:24:in `_app_views_requests_search_html_haml__1406362357321004239_70224614497480'
actionpack (4.0.3) lib/action_view/template.rb:143:in `block in render'
activesupport (4.0.3) lib/active_support/notifications.rb:161:in `instrument'
actionpack (4.0.3) lib/action_view/template.rb:141:in `render'
actionpack (4.0.3) lib/action_view/renderer/template_renderer.rb:49:in `block (2 levels) in render_template'
actionpack (4.0.3) lib/action_view/renderer/abstract_renderer.rb:38:in `block in instrument'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.0.3) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.0.3) lib/action_view/renderer/abstract_renderer.rb:38:in `instrument'
actionpack (4.0.3) lib/action_view/renderer/template_renderer.rb:48:in `block in render_template'
actionpack (4.0.3) lib/action_view/renderer/template_renderer.rb:56:in `render_with_layout'
actionpack (4.0.3) lib/action_view/renderer/template_renderer.rb:47:in `render_template'
actionpack (4.0.3) lib/action_view/renderer/template_renderer.rb:17:in `render'
actionpack (4.0.3) lib/action_view/renderer/renderer.rb:42:in `render_template'
actionpack (4.0.3) lib/action_view/renderer/renderer.rb:23:in `render'
actionpack (4.0.3) lib/abstract_controller/rendering.rb:127:in `_render_template'
actionpack (4.0.3) lib/action_controller/metal/streaming.rb:219:in `_render_template'
actionpack (4.0.3) lib/abstract_controller/rendering.rb:120:in `render_to_body'
actionpack (4.0.3) lib/action_controller/metal/rendering.rb:33:in `render_to_body'
actionpack (4.0.3) lib/action_controller/metal/renderers.rb:26:in `render_to_body'
actionpack (4.0.3) lib/abstract_controller/rendering.rb:97:in `render'
actionpack (4.0.3) lib/action_controller/metal/rendering.rb:16:in `render'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:41:in `block (2 levels) in render'
activesupport (4.0.3) lib/active_support/core_ext/benchmark.rb:12:in `block in ms'
/Users/pioz/.rvm/rubies/ruby-2.0.0-p353/lib/ruby/2.0.0/benchmark.rb:296:in `realtime'
activesupport (4.0.3) lib/active_support/core_ext/benchmark.rb:12:in `ms'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:41:in `block in render'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:84:in `cleanup_view_runtime'
activerecord (4.0.3) lib/active_record/railties/controller_runtime.rb:25:in `cleanup_view_runtime'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:40:in `render'
app/controllers/requests_controller.rb:31:in `block (2 levels) in results'
actionpack (4.0.3) lib/action_controller/metal/mime_responds.rb:191:in `call'
actionpack (4.0.3) lib/action_controller/metal/mime_responds.rb:191:in `respond_to'
app/controllers/requests_controller.rb:30:in `results'
actionpack (4.0.3) lib/action_controller/metal/implicit_render.rb:4:in `send_action'
actionpack (4.0.3) lib/abstract_controller/base.rb:189:in `process_action'
actionpack (4.0.3) lib/action_controller/metal/rendering.rb:10:in `process_action'
actionpack (4.0.3) lib/abstract_controller/callbacks.rb:18:in `block in process_action'
activesupport (4.0.3) lib/active_support/callbacks.rb:493:in `_run__2523755072312804404__process_action__callbacks'
activesupport (4.0.3) lib/active_support/callbacks.rb:80:in `run_callbacks'
actionpack (4.0.3) lib/abstract_controller/callbacks.rb:17:in `process_action'
actionpack (4.0.3) lib/action_controller/metal/rescue.rb:29:in `process_action'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:31:in `block in process_action'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `block in instrument'
activesupport (4.0.3) lib/active_support/notifications/instrumenter.rb:20:in `instrument'
activesupport (4.0.3) lib/active_support/notifications.rb:159:in `instrument'
actionpack (4.0.3) lib/action_controller/metal/instrumentation.rb:30:in `process_action'
actionpack (4.0.3) lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
activerecord (4.0.3) lib/active_record/railties/controller_runtime.rb:18:in `process_action'
actionpack (4.0.3) lib/abstract_controller/base.rb:136:in `process'
actionpack (4.0.3) lib/abstract_controller/rendering.rb:44:in `process'
actionpack (4.0.3) lib/action_controller/metal.rb:195:in `dispatch'
actionpack (4.0.3) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
actionpack (4.0.3) lib/action_controller/metal.rb:231:in `block in action'
actionpack (4.0.3) lib/action_dispatch/routing/route_set.rb:80:in `call'
actionpack (4.0.3) lib/action_dispatch/routing/route_set.rb:80:in `dispatch'
actionpack (4.0.3) lib/action_dispatch/routing/route_set.rb:48:in `call'
actionpack (4.0.3) lib/action_dispatch/journey/router.rb:71:in `block in call'
actionpack (4.0.3) lib/action_dispatch/journey/router.rb:59:in `each'
actionpack (4.0.3) lib/action_dispatch/journey/router.rb:59:in `call'
actionpack (4.0.3) lib/action_dispatch/routing/route_set.rb:680:in `call'
omniauth (1.2.1) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.1) lib/omniauth/strategy.rb:164:in `call'
omniauth (1.2.1) lib/omniauth/strategy.rb:186:in `call!'
omniauth (1.2.1) lib/omniauth/strategy.rb:164:in `call'
bullet (4.8.0) lib/bullet/rack.rb:10:in `call'
warden (1.2.3) lib/warden/manager.rb:35:in `block in call'
warden (1.2.3) lib/warden/manager.rb:34:in `catch'
warden (1.2.3) lib/warden/manager.rb:34:in `call'
rack (1.5.2) lib/rack/etag.rb:23:in `call'
rack (1.5.2) lib/rack/conditionalget.rb:35:in `call'
rack (1.5.2) lib/rack/head.rb:11:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/params_parser.rb:27:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/flash.rb:241:in `call'
rack (1.5.2) lib/rack/session/abstract/id.rb:225:in `context'
rack (1.5.2) lib/rack/session/abstract/id.rb:220:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/cookies.rb:486:in `call'
activerecord (4.0.3) lib/active_record/query_cache.rb:36:in `call'
activerecord (4.0.3) lib/active_record/connection_adapters/abstract/connection_pool.rb:626:in `call'
activerecord (4.0.3) lib/active_record/migration.rb:369:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (4.0.3) lib/active_support/callbacks.rb:373:in `_run__465812829959738644__call__callbacks'
activesupport (4.0.3) lib/active_support/callbacks.rb:80:in `run_callbacks'
actionpack (4.0.3) lib/action_dispatch/middleware/callbacks.rb:27:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/reloader.rb:64:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/remote_ip.rb:76:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
railties (4.0.3) lib/rails/rack/logger.rb:38:in `call_app'
railties (4.0.3) lib/rails/rack/logger.rb:20:in `block in call'
activesupport (4.0.3) lib/active_support/tagged_logging.rb:67:in `block in tagged'
activesupport (4.0.3) lib/active_support/tagged_logging.rb:25:in `tagged'
activesupport (4.0.3) lib/active_support/tagged_logging.rb:67:in `tagged'
railties (4.0.3) lib/rails/rack/logger.rb:20:in `call'
quiet_assets (1.0.2) lib/quiet_assets.rb:18:in `call_with_quiet_assets'
actionpack (4.0.3) lib/action_dispatch/middleware/request_id.rb:21:in `call'
rack (1.5.2) lib/rack/methodoverride.rb:21:in `call'
rack (1.5.2) lib/rack/runtime.rb:17:in `call'
activesupport (4.0.3) lib/active_support/cache/strategy/local_cache.rb:83:in `call'
rack (1.5.2) lib/rack/lock.rb:17:in `call'
actionpack (4.0.3) lib/action_dispatch/middleware/static.rb:64:in `call'
rack (1.5.2) lib/rack/sendfile.rb:112:in `call'
railties (4.0.3) lib/rails/engine.rb:511:in `call'
railties (4.0.3) lib/rails/application.rb:97:in `call'
rack (1.5.2) lib/rack/content_length.rb:14:in `call'
thin (1.6.1) lib/thin/connection.rb:82:in `block in pre_process'
thin (1.6.1) lib/thin/connection.rb:80:in `catch'
thin (1.6.1) lib/thin/connection.rb:80:in `pre_process'
thin (1.6.1) lib/thin/connection.rb:55:in `process'
thin (1.6.1) lib/thin/connection.rb:41:in `receive_data'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run_machine'
eventmachine (1.0.3) lib/eventmachine.rb:187:in `run'
thin (1.6.1) lib/thin/backends/base.rb:73:in `start'
thin (1.6.1) lib/thin/server.rb:162:in `start'
rack (1.5.2) lib/rack/handler/thin.rb:16:in `run'
rack (1.5.2) lib/rack/server.rb:264:in `start'
railties (4.0.3) lib/rails/commands/server.rb:84:in `start'
railties (4.0.3) lib/rails/commands.rb:76:in `block in <top (required)>'
railties (4.0.3) lib/rails/commands.rb:71:in `tap'
railties (4.0.3) lib/rails/commands.rb:71:in `<top (required)>'
bin/rails:4:in `require'
bin/rails:4:in `<main>'

When I run rake ts:generate I get:

rake ts:generate
Generating index files for post_core
rake aborted!
no such index 'post_core'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/connection.rb:90:in `rescue in query'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/connection.rb:93:in `query'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/connection.rb:71:in `execute'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/real_time/transcriber.rb:17:in `block in copy'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/connection.rb:37:in `block in take'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/innertube-1.1.0/lib/innertube.rb:138:in `take'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/connection.rb:35:in `take'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/real_time/transcriber.rb:16:in `copy'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/real_time/populator.rb:16:in `block in populate'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/relation/batches.rb:26:in `block (2 levels) in find_each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/relation/batches.rb:26:in `each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/relation/batches.rb:26:in `block in find_each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/relation/batches.rb:75:in `find_in_batches'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-deprecated_finders-1.0.3/lib/active_record/deprecated_finders/relation.rb:70:in `find_in_batches'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/relation/batches.rb:25:in `find_each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/activerecord-4.0.3/lib/active_record/querying.rb:8:in `find_each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/real_time/populator.rb:15:in `populate'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/real_time/populator.rb:3:in `populate'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/rake_interface.rb:19:in `block in generate'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/rake_interface.rb:18:in `each'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/rake_interface.rb:18:in `generate'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/gems/thinking-sphinx-3.1.0/lib/thinking_sphinx/tasks.rb:23:in `block (2 levels) in <top (required)>'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `eval'
/Users/pioz/.rvm/gems/ruby-2.0.0-p353/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => ts:generate
(See full trace by running task with --trace)

[pat]

Can you try rake ts:regenerate instead? Best to use that when changing index types.

[pioz]

Same output of rake ts:generate

[pat]

Can you manually stop all searchd processes and delete all sphinx files from db/sphinx/development and then run it again?

[pioz]

pyroblast:timerepublik2 pioz$ rake ts:stop
Stopped searchd daemon (pid: 20824).
pyroblast:timerepublik2 pioz$ rake ts:clear
pyroblast:timerepublik2 pioz$ rake ts:regenerate
searchd is not currently running.
Stopped searchd daemon (pid: ).
Generating configuration to /Users/pioz/Code/timerepublik2/config/development.sphinx.conf
Started searchd successfully (pid: 20841).
Generating index files for post_core
rake aborted!
no such index 'post_core'

[pat]

Does post_core exist in the generated Sphinx configuration file?

[pioz]

my development.sphinx.conf

indexer
{
  mem_limit = 1024M
}

searchd
{
  listen = 127.0.0.1:9301:mysql41
  log = /Users/pioz/Code/timerepublik2/log/development.searchd.log
  query_log = /Users/pioz/Code/timerepublik2/log/development.searchd.query.log
  pid_file = /Users/pioz/Code/timerepublik2/log/development.sphinx.pid
  workers = threads
  binlog_path = /Users/pioz/Code/timerepublik2/tmp/binlog/development
}

index post_core
{
  type = rt
  path = /Users/pioz/Code/timerepublik2/db/sphinx/development/post_core
  docinfo = extern
  charset_type = utf-8
  min_prefix_len = 3
  rt_field = sphinx_internal_class_name
  rt_field = type
  rt_field = first_name
  rt_field = last_name
  rt_field = talent_name
  rt_field = talent_t_name
  rt_field = message
  rt_attr_uint = sphinx_internal_id
  rt_attr_uint = sphinx_deleted
  rt_attr_uint = trustmeter
  rt_attr_uint = talent_id
  rt_attr_uint = user_id
  rt_attr_uint = banned
  rt_attr_uint = deactivated
  rt_attr_float = latitude
  rt_attr_float = longitude
  rt_attr_timestamp = created_at
  rt_attr_timestamp = registration_completed_at
  rt_attr_string = sphinx_internal_class
  rt_attr_string = first_name_sort
  rt_attr_string = last_name_sort
  rt_attr_string = talent_t_name_sort
}

source user_core_0
{
  type = mysql
  sql_host = localhost
  sql_user = root
  sql_pass = 
  sql_db = timerepublik2
  sql_query_pre = SET TIME_ZONE = '+0:00'
  sql_query = SELECT SQL_NO_CACHE `users`.`id` * 5 + 1 AS `id`, 'User' AS `sphinx_internal_class_name`, `users`.`first_name` AS `first_name`, `users`.`last_name` AS `last_name`, GROUP_CONCAT(DISTINCT talents.`name` SEPARATOR ' ') AS `talent_name`, GROUP_CONCAT(DISTINCT talent_translations.`name` SEPARATOR ' ') AS `talent_t_name`, GROUP_CONCAT(DISTINCT tags_users.`list` SEPARATOR ' ') AS `tag_list`, `users`.`id` AS `sphinx_internal_id`, 'User' AS `sphinx_internal_class`, 0 AS `sphinx_deleted`, RADIANS(users.latitude) AS `latitude`, RADIANS(users.longitude) AS `longitude`, `users`.`trustmeter` AS `trustmeter`, UNIX_TIMESTAMP(`users`.`created_at`) AS `created_at`, GROUP_CONCAT(DISTINCT tags_users.`talent_id` SEPARATOR ',') AS `talent_id`, GROUP_CONCAT(DISTINCT tags_users.`id` SEPARATOR ',') AS `tag_id`, `users`.`inverse_favoritings_count` AS `favorites` FROM `users` LEFT OUTER JOIN `tags` ON `tags`.`user_id` = `users`.`id` LEFT OUTER JOIN `talents` ON `talents`.`id` = `tags`.`talent_id` LEFT OUTER JOIN `talent_translations` ON `talent_translations`.`talent_id` = `talents`.`id` LEFT OUTER JOIN `tags` `tags_users` ON `tags_users`.`user_id` = `users`.`id` WHERE (`users`.`id` BETWEEN $start AND $end AND users.banned = 0 AND users.deactivated = 0 AND users.registration_completed_at IS NOT NULL AND users.id > 0) GROUP BY `users`.`id`, `users`.`first_name`, `users`.`last_name`, `users`.`id`, `users`.`trustmeter`, `users`.`created_at`, `users`.`inverse_favoritings_count`  ORDER BY NULL
  sql_query_range = SELECT IFNULL(MIN(`users`.`id`), 1), IFNULL(MAX(`users`.`id`), 1) FROM `users`
  sql_range_step = 10000000
  sql_attr_uint = sphinx_internal_id
  sql_attr_uint = sphinx_deleted
  sql_attr_uint = trustmeter
  sql_attr_uint = favorites
  sql_attr_timestamp = created_at
  sql_attr_float = latitude
  sql_attr_float = longitude
  sql_attr_multi = uint talent_id from field
  sql_attr_multi = uint tag_id from field
  sql_attr_string = sphinx_internal_class
  sql_field_string = first_name
  sql_field_string = last_name
  sql_query_post_index = UPDATE `users` SET `delta` = 0 WHERE `delta` = 1
  sql_query_info = SELECT `users`.* FROM `users`  WHERE (`users`.`id` = ($id - 1) / 5)
}

index user_core
{
  type = plain
  path = /Users/pioz/Code/timerepublik2/db/sphinx/development/user_core
  docinfo = extern
  charset_type = utf-8
  min_prefix_len = 3
  source = user_core_0
}

source user_delta_0
{
  type = mysql
  sql_host = localhost
  sql_user = root
  sql_pass = 
  sql_db = timerepublik2
  sql_query_pre = SET TIME_ZONE = '+0:00'
  sql_query = SELECT SQL_NO_CACHE `users`.`id` * 5 + 1 AS `id`, 'User' AS `sphinx_internal_class_name`, `users`.`first_name` AS `first_name`, `users`.`last_name` AS `last_name`, GROUP_CONCAT(DISTINCT talents.`name` SEPARATOR ' ') AS `talent_name`, GROUP_CONCAT(DISTINCT talent_translations.`name` SEPARATOR ' ') AS `talent_t_name`, GROUP_CONCAT(DISTINCT tags_users.`list` SEPARATOR ' ') AS `tag_list`, `users`.`id` AS `sphinx_internal_id`, 'User' AS `sphinx_internal_class`, 0 AS `sphinx_deleted`, RADIANS(users.latitude) AS `latitude`, RADIANS(users.longitude) AS `longitude`, `users`.`trustmeter` AS `trustmeter`, UNIX_TIMESTAMP(`users`.`created_at`) AS `created_at`, GROUP_CONCAT(DISTINCT tags_users.`talent_id` SEPARATOR ',') AS `talent_id`, GROUP_CONCAT(DISTINCT tags_users.`id` SEPARATOR ',') AS `tag_id`, `users`.`inverse_favoritings_count` AS `favorites` FROM `users` LEFT OUTER JOIN `tags` ON `tags`.`user_id` = `users`.`id` LEFT OUTER JOIN `talents` ON `talents`.`id` = `tags`.`talent_id` LEFT OUTER JOIN `talent_translations` ON `talent_translations`.`talent_id` = `talents`.`id` LEFT OUTER JOIN `tags` `tags_users` ON `tags_users`.`user_id` = `users`.`id` WHERE (`users`.`delta` = 1 AND `users`.`id` BETWEEN $start AND $end AND users.banned = 0 AND users.deactivated = 0 AND users.registration_completed_at IS NOT NULL AND users.id > 0) GROUP BY `users`.`id`, `users`.`first_name`, `users`.`last_name`, `users`.`id`, `users`.`trustmeter`, `users`.`created_at`, `users`.`inverse_favoritings_count`  ORDER BY NULL
  sql_query_range = SELECT IFNULL(MIN(`users`.`id`), 1), IFNULL(MAX(`users`.`id`), 1) FROM `users`  WHERE (`users`.`delta` = 1)
  sql_range_step = 10000000
  sql_attr_uint = sphinx_internal_id
  sql_attr_uint = sphinx_deleted
  sql_attr_uint = trustmeter
  sql_attr_uint = favorites
  sql_attr_timestamp = created_at
  sql_attr_float = latitude
  sql_attr_float = longitude
  sql_attr_multi = uint talent_id from field
  sql_attr_multi = uint tag_id from field
  sql_attr_string = sphinx_internal_class
  sql_field_string = first_name
  sql_field_string = last_name
  sql_query_info = SELECT `users`.* FROM `users`  WHERE (`users`.`id` = ($id - 1) / 5)
}

index user_delta
{
  type = plain
  path = /Users/pioz/Code/timerepublik2/db/sphinx/development/user_delta
  docinfo = extern
  charset_type = utf-8
  min_prefix_len = 3
  source = user_delta_0
}

index post
{
  type = distributed
  local = post_core
}

index user
{
  type = distributed
  local = user_core
  local = user_delta
}

[pat]

Can you change your user index to be a real_time index as well, just in case it's the mix of the two that's causing problems?

[pioz]

Same problem. But in my development.searchd.log I have:

[Wed Feb 26 00:37:32.446 2014] [21054] Child process 21055 has been forked
[Wed Feb 26 00:37:32.447 2014] [21055] listening on 127.0.0.1:9301
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: ERROR: index 'post_core': RT indexes support prefixes and infixes with only dict=keywords - NOT SERVING
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: ERROR: index 'user_core': RT indexes support prefixes and infixes with only dict=keywords - NOT SERVING
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: index 'post': no such local index 'post_core' - SKIPPING LOCAL INDEX
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: index 'post': no valid local/remote indexes in distributed index - NOT SERVING
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: index 'user': no such local index 'user_core' - SKIPPING LOCAL INDEX
[Wed Feb 26 00:37:32.447 2014] [21055] WARNING: index 'user': no valid local/remote indexes in distributed index - NOT SERVING
[Wed Feb 26 00:37:32.447 2014] [21055] FATAL: no valid indexes to serve
[Wed Feb 26 00:37:32.448 2014] [21054] last message repeated 1 times
[Wed Feb 26 00:37:32.448 2014] [21054] Child process 21055 has been finished, exit code 1. Watchdog finishes also. Good bye!

[pat]

Ah! Add dict: keywords to your config/thinking_sphinx.yml file.

Really wish Sphinx raised that error when starting the daemon, instead of just noting it in the logs.

[pioz]

Wowowowowowow, now works! I'll test this configuration... thx for your help. I'll vote you on http://rubyheroes.com/

[pat]

Hahah, I appreciate the vote - but I've actually already received a Ruby Hero award (in 2009).

[pioz]

-.-

[pioz]

Ok, I'm trying to figure out with real time indices...

Here my user index:

ThinkingSphinx::Index.define :user, with: :real_time do
  indexes first_name, sortable: :insensitive
  indexes last_name, sortable: :insensitive
  # add first_name again (sortable option seems do not work)
  has first_name, as: :name, type: :string
end

in user.rb I've:

class User < ActiveRecord::Base
  ...
  after_save do
    ThinkingSphinx::RealTime.callback_for(:user)
  end
end

Now if I search with User.search(order: 'name ASC') all work. If I edit the user first_name field the sorting do not change without launch rake ts:reconfigure.

Some notes:

• The option sortable: :insensitive in index does not seem to work. I need to use the has method.
• I must specify all types in my indices definition.

[pat]

Editing the first_name should be reflected in Sphinx with sorting... so that's certainly annoying if that's not the case. Can you run through an example here of what you're changing and the search results that aren't taking the change into account?

Yes, all types are required - because it's not database-backed, there's no way to detect types automatically.

sortable: :insensitive doesn't work - I don't think that was brought across from TS v1/v2 - but that was from when Sphinx string attributes weren't properly sortable. They now are, so I'd expect the default to be case-insensitive anyway, given that's what Sphinx defaults to: http://sphinxsearch.com/docs/manual-2.1.5.html#conf-collation-server

[pioz]

This is a example rails app: https://github.com/pioz/thinking_sphinx_real_time_indices_example

When I edit an user the sorting do not change.

[pat]

Many thanks - made it much easier to debug.

The issue is how you've set up the callback - the TS code returns something for the callback to run, instead of running that code within a callback. So, change it to the following and it'll work:

after_save ThinkingSphinx::RealTime.callback_for(:user)

[pioz]

Great, this works. Now I make stuff a little bit complicated. In our example (https://github.com/pioz/thinking_sphinx_real_time_indices_example) I have added a one to many association: user has_many talents, talent belongs_to user.

In the user index I've added the talent name:

ThinkingSphinx::Index.define :user, with: :real_time do
  indexes first_name
  indexes talents.name, as: :talent_name
  has first_name, as: :name, type: :string
end

Now if I search with a talent name (I've added a form to search users in users#index), Sphinx returns no users. Why? Is the index definition wrong? Also searching with star does not seem to work.

To update the example:

git pull
bundle
rake db:migrate
rake db:fixtures:load
rake ts:regenerate
rails s

[pioz]

I've solved with star search. I need to add

enable_star: true
min_prefix_len: 3

in thinking_sphinx.yml config file and search with User.search('foobar', star: true)

[pioz]

Ok, for search through associations I figure out indexing in this way:

ThinkingSphinx::Index.define :user, with: :real_time do
  indexes first_name
  indexes talents.map(&:name).join(' '), as: :talent_name
  has first_name, as: :name, type: :string
end

Is this the right way to proceed? For geoloc I can do something like this:

ThinkingSphinx::Index.define :user, with: :real_time do
  indexes first_name
  indexes talents.map(&:name).join(' '), as: :talent_name
  has first_name, as: :name, type: :string
  has latitude_in_radiant, as: :latitude, type: :float
  has longitude_in_radiant, as: :longitude, type: :float
end

class User < ActiveRecord::Base
  ...
  def latitude_in_radiant
    self.latitude * Math::PI / 180.0
  end
  def longitude_in_radiant
    self.longitude * Math::PI / 180.0
  end

end

[pat]

Your approach for lat/lng is spot on. As for talents, you want a method in your User model that does the collecting for you.

# in the index
indexes talent_names

# in the model
def talent_names
  talents.map(&:name).join(' ')
end

[pioz]

Type multi for real time indices is supported?

Im trying to convert this:

ThinkingSphinx::Index.define :user, with: :active_record, delta: true do
  indexes first_name
  has tags.talent_id, as: :talent_ids
end

User.search '', with: { talent_ids: x }

to

ThinkingSphinx::Index.define :user, with: :real_time do
  indexes first_name
  has talent_ids, type: :multi
end

User.search '', with: { talent_ids: x }

But I get the error Unknown attribute type 'multi'

Is there a way to do this with real time indeces?

[pioz]

Oh, ok I can do with:

  has talent_ids, type: :integer, multi: true

[pat]

Exactly :)

[pioz]

Hi again. In our example (https://github.com/pioz/thinking_sphinx_real_time_indices_example) I have added a new association: user has many followers that are users. Now in users_path I show all users order by the number of followers (this count is saved in the users table with the counter cache followers_count). If you edit a user adding some users to follow the sort is wrong if followers_count is 1 (first time that a user has a follower). If followers_count is 2 or more the order return to be correct.

Can you check the example?

[pioz]

I've check the code of thinking sphinx. Seems that the objects loaded by the method objects_for in file real_time_callbacks.rb line 28 are not updated with counter cache. If I modify the method in this way:

  def objects_for(instance)
    Array(path.inject(instance) { |object, method| object.send(method).reload })
  end

all works.

[pat]

(I realise this is a month old - sorry!)

I'm not a fan of adding in the reload call and thus hitting the database for what will almost certainly be unnecessary in the majority of cases (i.e. anything without counter caches). I guess the issue here is that counter caches are updated outside of standard model workflows, and thus the model itself doesn't change (only the database does). If it's outside the standard workflow, then callbacks don't get run, and thus the updates aren't happening until something else fires them off.

I guess what you could do is manually call an update callback for actions that involve updating counter caches?

# Use the arguments to this method call as appropriate for your setup
callback = ThinkingSphinx::RealTime.callback_for(:product)
# And then invoke it for a specific instance - and make sure that instance has the latest data:
callback.after_save(instance.reload)

[pioz]

Ok, this works!

[jogaco]

Guys. I spent some time until found out why my real_time indices where not working. Until I found-out your comment about dict: keywords in the config!

[pioz]

:+1:

[hafsabatool7]

I tried to use default_scope this way but it does not work. @results = ThinkingSphinx.search( params[:search], classes: [SaleOrder, PurchaseOrder, Medicine, Doctor, Staff, Patient, Department] )

I tried calling search like this. However, even when I have default scope specified in each of these models/classes, it won't work. PLEASE HELP!

[pat]

Hi @hafsabatool7 - as noted in an early comment in this thread default ActiveRecord scopes aren't used in search calls because searches aren't SQL calls to your database - they're instead SphinxQL calls to Sphinx.

There is the option of having Sphinx scopes - however, these apply only on single-model searches. Your example is searching across many models, and thus there's no central source for scopes - a search on many models is a single query to Sphinx, not separate queries per model, so combining the scopes from different models into the single query is not possible.

[hafsabatool7]

@pat Thank you for your response.