Prefetch tasks git file revision info resolves #1248

[MontrealSergiy]

Prefetches tasks git file revision info #1248

Note, this will fail e.g. if there are ghost tools (configs) from deleted boutiques.

I can add some rescues if that's an issue, or just clean up the db from outdated configs

[natacha-beck]

@MontrealSergiy works on my side reduce the execution of git command.

But you said:

Note, this will fail e.g. if there are ghost tools (configs) from deleted boutiques.

How it will failed, what's happen in this specific case.

[MontrealSergiy]

In a inconsistent db the feature can put down cbrain workers with exception NameError while constantizing. But purging db and reinstall of plugins helps. The exception, as the cause of worker death only reported in logs, w.r.t GUI a worker dies silently

BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:270:in `const_get'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:270:in `block in constantize'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `each'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `inject'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/inflector/methods.rb:266:in `constantize'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/core_ext/string/inflections.rb:66:in `constantize'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/tool.rb:74:in `cbrain_task_class'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/tool_config.rb:324:in `cbrain_task_class'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38:in `map'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/relation/delegation.rb:38:in `map'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/bourreau_worker.rb:60:in `setup'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:384:in `main_loop'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:291:in `block in start'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:142:in `block (2 levels) in spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:102:in `fork'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:102:in `block in spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:96:in `fork'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/config/initializers/cbrain.rb:96:in `spawn_with_active_records'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker.rb:204:in `start'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/worker_pool.rb:76:in `create_or_find_pool'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/bourreau.rb:410:in `process_command_start_workers'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/models/remote_resource.rb:786:in `process_command'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/controllers/controls_controller.rb:123:in `process_command'
BourreauWorker-20855: /home/users/sboroday/cbrain/Bourreau/app/controllers/controls_controller.rb:92:in `create'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/base.rb:188:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/rendering.rb:30:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:750:in `_run_process_action_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:90:in `run_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/callbacks.rb:19:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/rescue.rb:20:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications.rb:164:in `block in instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/notifications.rb:164:in `instrument'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal/params_wrapper.rb:248:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/abstract_controller/base.rb:126:in `process'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionview-5.0.7/lib/action_view/rendering.rb:30:in `process'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal.rb:190:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_controller/metal.rb:262:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:32:in `serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:39:in `block in serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:26:in `each'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/journey/router.rb:26:in `serve'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/routing/route_set.rb:727:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/cookies.rb:613:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.7/lib/active_record/migration.rb:553:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/callbacks.rb:90:in `run_callbacks'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/executor.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:36:in `call_app'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:24:in `block in call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:69:in `block in tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:26:in `tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/tagged_logging.rb:69:in `tagged'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/rack/logger.rb:24:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/request_id.rb:24:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/activesupport-5.0.7/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/executor.rb:12:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/actionpack-5.0.7/lib/action_dispatch/middleware/static.rb:136:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/railties-5.0.7/lib/rails/engine.rb:522:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/configuration.rb:225:in `call'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:632:in `handle_request'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:446:in `process_client'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/server.rb:306:in `block in run'
BourreauWorker-20855: /home/users/sboroday/.rvm/gems/ruby-2.4.1/gems/puma-3.11.4/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
BourreauWorker-20855: Worker shutting down.```

[MontrealSergiy]

a rescue can prevent that, smth like (ToolConfig.where(:bourreau_id => @rr_id).map {|tc| tc.cbrain_task_class rescue nil} ).uniq.each { |klass| klass&.revision_info&.self_update },

The problem is more likely occur in a dev environment, when tool/config get renamed etc. But if exists on production an admin can delete broken tools/config/tasks manually

[natacha-beck]

According to your commet:

In a inconsistent db the feature can put down cbrain workers with exception NameError > while constantizing. But purging db and reinstall of plugins helps. The exception, as the cause of worker death only reported in logs, w.r.t GUI a worker dies silently

You should add rescue we cannot let worker die without any message. We cannot let worker dies just for that.

[natacha-beck]

What about the following:

  ToolConfig.where(:bourreau_id => @rr_id)
            .map {|tc| tc.cbrain_task_class rescue nil}
            .uniq
            .compact   # to remove nil
            .each { |klass| klass.revision_info.self_update }

[MontrealSergiy]

A bit longer than what I suggested above yet should work too

[MontrealSergiy]

@prioux your comments addressed

[MontrealSergiy]

if you like I can rebase

[prioux]

OK I'll check this soon

[prioux]

This seems ok to me except for one minor thing: you created your method as a110 and in the file you put that method between a090 and a100. Just put it at the proper place please.

[MontrealSergiy]

This seems ok to me except for one minor thing: you created your method as a110 and in the file you put that method between a090 and a100. Just put it at the proper place please.

ok, the method is moved toward the end