search

ManageIQ / manageiq-providers-kubernetes

ManageIQ plugin for the Kubernetes provider.
https://kubernetes.io/
Apache License 2.0
7 stars 63 forks source link

PartialRefreshError: no block given (yield) during refresh #136

Closed moolitayer closed 7 years ago

moolitayer commented 7 years ago

2.4.1 :002 > EmsRefresh.refresh(ManageIQ::Providers::ContainerManager.first)

# Output from evm log after scheduled refresh
[----] E, [2017-10-23T11:57:08.992318 #23005:854f80] ERROR -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) EMS: [48.11], id: [10] Unable to perform refresh for the following targets:
[----] E, [2017-10-23T11:57:08.992362 #23005:854f80] ERROR -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh)  --- ManageIQ::Providers::Openshift::ContainerManager [48.11] id [10]
[----] I, [2017-10-23T11:57:09.009942 #23005:854f80]  INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) Refreshing all targets...Complete
[----] E, [2017-10-23T11:57:09.010147 #23005:854f80] ERROR -- : MIQ(MiqQueue#deliver) Message id: [24640], Error: [no block given (yield)]
[----] E, [2017-10-23T11:57:09.010374 #23005:854f80] ERROR -- : [EmsRefresh::Refreshers::EmsRefresherMixin::PartialRefreshError]: no block given (yield)  Method:[block in method_missing]
[----] E, [2017-10-23T11:57:09.010611 #23005:854f80] ERROR -- : /home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:50:in `refresh'
/home/mtayer/dev/manageiq/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:97:in `block in refresh'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:96:in `each'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:96:in `refresh'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:452:in `block in dispatch_method'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:451:in `dispatch_method'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:428:in `block in deliver'
/home/mtayer/dev/manageiq/app/models/user.rb:252:in `with_user_group'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:428:in `deliver'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:105:in `deliver_queue_message'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:331:in `block in do_work_loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:328:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:328:in `do_work_loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:155:in `run'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:129:in `start'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:22:in `start_worker'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:354:in `block in start_runner_via_fork'
/home/mtayer/.rvm/gems/ruby-2.4.1/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/mtayer/.rvm/gems/ruby-2.4.1/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:352:in `start_runner_via_fork'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:346:in `start_runner'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:380:in `start'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:263:in `start_worker'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:327:in `block in monitor'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-b956c0636c7f/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-b956c0636c7f/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:327:in `monitor'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:349:in `block (2 levels) in monitor_loop'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-b956c0636c7f/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-b956c0636c7f/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:349:in `block in monitor_loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:348:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:348:in `monitor_loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:232:in `start'
/home/mtayer/dev/manageiq/lib/workers/evm_server.rb:27:in `start'
/home/mtayer/dev/manageiq/lib/workers/evm_server.rb:48:in `start'
/home/mtayer/dev/manageiq/lib/workers/bin/evm_server.rb:4:in `<main>'
moolitayer commented 7 years ago

cc @cben have you seen this?

cben commented 7 years ago

Haven't seen. no block given (yield) Method:[block in method_missing] sounds weird. Stacktrace not informative, this is just https://github.com/ManageIQ/manageiq/blob/2ec76f1244/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb#L27-L50 swallowing the exception. @moolitayer Can you set EmsRefresh.debug_failures = true and re-run?

moolitayer commented 7 years ago

Maybe this is more relevant? (I'm not sure where this is coming from, did you get a chance to add a new provider and refresh it recently?)

[----] I, [2017-10-25T15:36:11.151320 #24803:3bef84]  INFO -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh_targets_for_ems) EMS: [48.11], id: [12] Refreshing target ManageIQ::Provide
rs::Openshift::ContainerManager [48.11] id [12]...
[----] E, [2017-10-25T15:36:11.151608 #24803:3bef84] ERROR -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::Refresher#refresh) EMS: [48.11], id: [12] Refresh failed
[----] E, [2017-10-25T15:36:11.151799 #24803:3bef84] ERROR -- : [LocalJumpError]: no block given (yield)  Method:[block in method_missing]
[----] E, [2017-10-25T15:36:11.151964 #24803:3bef84] ERROR -- : /home/mtayer/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.6/lib/active_record/connection_adapters/abstract/query_cache.rb:56:in `cache'
/home/mtayer/.rvm/gems/ruby-2.4.1/gems/activerecord-5.0.6/lib/active_record/query_cache.rb:9:in `cache'
/home/mtayer/dev/manageiq/plugins/manageiq-providers-kubernetes/app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb:21:in `initialize'
/home/mtayer/dev/manageiq/plugins/manageiq-providers-kubernetes/app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb:13:in `new'
/home/mtayer/dev/manageiq/plugins/manageiq-providers-kubernetes/app/models/manageiq/providers/kubernetes/container_manager/refresh_parser.rb:13:in `ems_inv_to_inv_collections'
/home/mtayer/dev/manageiq/plugins/manageiq-providers-openshift/app/models/manageiq/providers/openshift/container_manager/refresher.rb:33:in `parse_targeted_inventory'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:88:in `block in refresh_targets_for_ems'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:87:in `refresh_targets_for_ems'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block (2 levels) in refresh'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:24:in `block in refresh'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `each'
/home/mtayer/dev/manageiq/app/models/ems_refresh/refreshers/ems_refresher_mixin.rb:14:in `refresh'
/home/mtayer/dev/manageiq/app/models/manageiq/providers/base_manager/refresher.rb:9:in `refresh'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:97:in `block in refresh'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:96:in `each'
/home/mtayer/dev/manageiq/app/models/ems_refresh.rb:96:in `refresh'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:449:in `block in dispatch_method'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `block in catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:33:in `catch'
/home/mtayer/.rvm/rubies/ruby-2.4.1/lib/ruby/2.4.0/timeout.rb:108:in `timeout'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:448:in `dispatch_method'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:425:in `block in deliver'
/home/mtayer/dev/manageiq/app/models/user.rb:252:in `with_user_group'
/home/mtayer/dev/manageiq/app/models/miq_queue.rb:425:in `deliver'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:105:in `deliver_queue_message'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:134:in `deliver_message'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:152:in `block in do_work'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_queue_worker_base/runner.rb:146:in `do_work'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:331:in `block in do_work_loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:328:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:328:in `do_work_loop'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:155:in `run'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:129:in `start'
/home/mtayer/dev/manageiq/app/models/miq_worker/runner.rb:22:in `start_worker'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:354:in `block in start_runner_via_fork'
/home/mtayer/.rvm/gems/ruby-2.4.1/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/mtayer/.rvm/gems/ruby-2.4.1/gems/nakayoshi_fork-0.0.3/lib/nakayoshi_fork.rb:24:in `fork'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:352:in `start_runner_via_fork'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:346:in `start_runner'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:380:in `start'
/home/mtayer/dev/manageiq/app/models/miq_worker.rb:263:in `start_worker'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:68:in `start_worker_for_ems'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:46:in `block in sync_workers'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `each'
/home/mtayer/dev/manageiq/app/models/mixins/per_ems_worker_mixin.rb:45:in `sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:53:in `block in sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `each'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:50:in `sync_workers'
/home/mtayer/dev/manageiq/app/models/miq_server/worker_management/monitor.rb:22:in `monitor_workers'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:327:in `block in monitor'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:28:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:327:in `monitor'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:349:in `block (2 levels) in monitor_loop'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:11:in `realtime_store'
/home/mtayer/.rvm/gems/ruby-2.4.1/bundler/gems/manageiq-gems-pending-93e11d4fe973/lib/gems/pending/util/extensions/miq-benchmark.rb:35:in `realtime_block'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:349:in `block in monitor_loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:348:in `loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:348:in `monitor_loop'
/home/mtayer/dev/manageiq/app/models/miq_server.rb:232:in `start'
/home/mtayer/dev/manageiq/lib/workers/evm_server.rb:27:in `start'
/home/mtayer/dev/manageiq/lib/workers/evm_server.rb:48:in `start'
/home/mtayer/dev/manageiq/lib/workers/bin/evm_server.rb:4:in `<main>'
cben commented 7 years ago

Can you rebase, both k8s provider and core manageiq? On master kubernetes/container_manager/refresh_parser.rb:21 calls ContainerLabelTagMapping.mapper, but until recently it was ContainerLabelTagMapping.cache. Theory: If you have old k8s (before #130) and new core, it'd call cache, no longer defined on the model, which explains where cache from active_record/query_cache.rb:9 comes in.

moolitayer commented 7 years ago

I pretty sure I tested master with no plugins, anyway it's now working fine. Thanks @cben