key not found: :action, Did you mean? :ction

[evgeni]

First of all, I have no idea what I did :disappointed_relieved:

I was poking around https://projects.theforeman.org/issues/34394, and restarting PostgreSQL and dynflow a lot, and at one point the orchestrator started refusing to start:

Feb 10 09:21:09 centos8-stream-katello-nightly.tanso.example.com systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Feb 10 09:21:16 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: 2022-02-10T09:21:16.827Z 716 TID-120 INFO: GitLab reliable fetch activated!
Feb 10 09:21:17 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: 2022-02-10T09:21:17.114Z 716 TID-110 INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-716", :url=>"redis:/>
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: key not found: :action
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: Did you mean?  :ction
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:289:in `fetch'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:289:in `table'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:332:in `save'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence_adapters/sequel.rb:178:in `save_action'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/persistence.rb:46:in `save_action'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/plan_step.rb:108:in `initialize_action'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:405:in `block in add_plan_step'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:400:in `tap'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:400:in `add_plan_step'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan.rb:279:in `prepare'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:218:in `block (2 levels) in plan_with_options'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/coordinator.rb:326:in `acquire'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:217:in `block in plan_with_options'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `tap'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:216:in `plan_with_options'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/active_job/queue_adapter.rb:9:in `block in enqueue'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/world.rb:178:in `trigger'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/active_job/queue_adapter.rb:7:in `enqueue'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/enqueuing.rb:59:in `block in enqueue'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `block in instrument'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in `instrument'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/logging.rb:35:in `block (2 levels) in <module:Logging>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `instance_exec'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/logging.rb:16:in `block (3 levels) in <module:Logging>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/logging.rb:47:in `tag_logger'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/logging.rb:15:in `block (2 levels) in <module:Logging>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `instance_exec'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in `run_callbacks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/enqueuing.rb:55:in `enqueue'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/activejob-6.0.3.7/lib/active_job/enqueuing.rb:22:in `perform_later'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/foreman/app/jobs/application_job.rb:24:in `spawn_if_missing'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/foreman/config/initializers/rss_notifications.rb:3:in `block in <top (required)>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails/configuration.rb:67:in `block in run_on_init_hooks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails/configuration.rb:67:in `each'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails/configuration.rb:67:in `run_on_init_hooks'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails.rb:43:in `block in initialize!'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails.rb:38:in `tap'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/rails.rb:38:in `initialize!'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/foreman/extras/dynflow-sidekiq.rb:34:in `<top (required)>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in `boot_system'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in `run'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in `<top (required)>'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/bin/sidekiq:23:in `load'
Feb 10 09:21:55 centos8-stream-katello-nightly.tanso.example.com dynflow-sidekiq@orchestrator[716]: /usr/bin/sidekiq:23:in `<main>'
Feb 10 09:21:56 centos8-stream-katello-nightly.tanso.example.com systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 09:21:56 centos8-stream-katello-nightly.tanso.example.com systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
Feb 10 09:21:56 centos8-stream-katello-nightly.tanso.example.com systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.

Here is a sosreport from the machine, and I can provide ssh access too.

sosreport-centos8-stream-katello-nightly-2022-02-10-mjpjjdq.tar.gz

[evgeni]

@evgeni was stupid :(