Closed fraenki closed 19 hours ago
Just out of memory
:exclamation: Be warned this will remove all your trends data, but it looks like you've none already.
You can run /usr/sbin/foreman-rake purge:trends
See implementation: https://github.com/theforeman/foreman/blob/e9891a9e3292e6755113164edd7c1d8dbb4648de/lib/tasks/cleanup.rake#L15
The important line is: https://github.com/theforeman/foreman/blob/e9891a9e3292e6755113164edd7c1d8dbb4648de/lib/cleanup_helper.rb#L18
That will remove all notes about trends and you can migrate the plugin again.
Unfortunately, this does not work either :-(
$ /usr/sbin/foreman-rake --trace purge:trends
** Invoke purge:trends (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute purge:trends
rake aborted!
Trends data could not be purged
/usr/share/foreman/lib/tasks/cleanup.rake:17:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Tasks: TOP => purge:trends
Oh, the error message has changed. Now I get the following error:
$ /usr/sbin/foreman-rake --trace purge:trends
(errors)
$ /usr/sbin/foreman-rake db:seed
$ /usr/sbin/foreman-rake db:migrate
(no output)
$ /usr/sbin/foreman-rake foreman_statistics:trends:counter
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "environment_id" does not exist
LINE 1: SELECT COUNT(*) AS count_all, "environment_id" AS environmen...
^
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `exec_query'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:532:in `select'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:69:in `select_all'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/query_cache.rb:103:in `select_all'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:364:in `block in execute_grouped_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation.rb:861:in `skip_query_cache_if_necessary'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:364:in `execute_grouped_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:267:in `perform_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:144:in `calculate'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:51:in `count'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:31:in `block in update_trend_counters'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:27:in `update_trend_counters'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:6:in `update!'
/usr/share/gems/gems/foreman_statistics-2.1.0/lib/tasks/foreman_statistics_tasks.rake:8:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Caused by:
PG::UndefinedColumn: ERROR: column "environment_id" does not exist
LINE 1: SELECT COUNT(*) AS count_all, "environment_id" AS environmen...
^
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `exec_query'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:532:in `select'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/database_statements.rb:69:in `select_all'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/connection_adapters/abstract/query_cache.rb:103:in `select_all'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:364:in `block in execute_grouped_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation.rb:861:in `skip_query_cache_if_necessary'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:364:in `execute_grouped_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:267:in `perform_calculation'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:144:in `calculate'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/calculations.rb:51:in `count'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:31:in `block in update_trend_counters'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7.7/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:27:in `update_trend_counters'
/usr/share/gems/gems/foreman_statistics-2.1.0/app/services/foreman_statistics/trend_importer.rb:6:in `update!'
/usr/share/gems/gems/foreman_statistics-2.1.0/lib/tasks/foreman_statistics_tasks.rake:8:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => foreman_statistics:trends:counter
(See full trace by running task with --trace)
Hmm can you run /usr/sbin/foreman-console
And just run
ActiveRecord::SchemaMigration.where(version: %w[20121012170851 20121012170936 20150202094307 20181031155025]).delete_all
And then try /usr/sbin/foreman-rake db:migrate
again?
This did the trick! Thank you :)
$ /usr/sbin/foreman-rake console
Loading production environment (Rails 6.1.7.7)
irb(main):001:0> ActiveRecord::SchemaMigration.where(version: %w[20121012170851 20121012170936 20150202094307 20181031155025]).delete_all
=> 4
irb(main):002:0>
$ /usr/sbin/foreman-rake db:migrate
== 20121012170851 CreateTrends: migrating =====================================
-- create_table(:trends, {})
-> 0.0927s
-- add_index(:trends, :type)
-> 0.0043s
-- add_index(:trends, [:trendable_type, :trendable_id])
-> 0.0024s
-- add_index(:trends, :fact_value)
-> 0.0036s
== 20121012170851 CreateTrends: migrated (0.1033s) ============================
== 20121012170936 CreateTrendCounters: migrating ==============================
-- create_table(:trend_counters, {})
-> 0.0291s
== 20121012170936 CreateTrendCounters: migrated (0.0292s) =====================
== 20150202094307 AddRangeToTrendCounters: migrating ==========================
-- add_column(:trend_counters, :interval_start, :datetime)
-> 0.0007s
-- add_column(:trend_counters, :interval_end, :datetime)
-> 0.0003s
== 20150202094307 AddRangeToTrendCounters: migrated (0.0011s) =================
== 20181031155025 AddTrendCounterCreatedAtUniqueConstraint: migrating =========
-- add_index(:trend_counters, [:trend_id, :created_at], {:unique=>true})
-> 0.0040s
== 20181031155025 AddTrendCounterCreatedAtUniqueConstraint: migrated (0.0041s)
Problem
Trends are broken for unknown reason. Is there any way to reset Trends or recover from this situation?
It looks like the database table does not exist (anymore):
Expected outcome
I’ve tried to seed the tables, but it does not recreate the tables:
$ /usr/sbin/foreman-rake db:seed $ /usr/sbin/foreman-rake db:migrate
Foreman and Proxy versions
3.10.0
Foreman and Proxy plugin versions: foreman-3.10.0-1.el8.noarch foreman-cli-3.10.0-1.el8.noarch foreman-debug-3.10.0-1.el8.noarch foreman-dynflow-sidekiq-3.10.0-1.el8.noarch foreman-obsolete-packages-1.6-1.el8.noarch foreman-ovirt-3.10.0-1.el8.noarch foreman-postgresql-3.10.0-1.el8.noarch foreman-proxy-3.10.0-1.el8.noarch foreman-redis-3.10.0-1.el8.noarch foreman-selinux-3.10.0-1.el8.noarch foreman-service-3.10.0-1.el8.noarch rubygem-foreman_discovery-24.0.1-1.fm3_10.el8.noarch rubygem-foreman_hooks-0.3.17-3.fm3_9.el8.noarch rubygem-foreman_puppet-6.2.0-1.fm3_10.el8.noarch rubygem-foreman_statistics-2.1.0-2.fm3_10.el8.noarch rubygem-foreman_templates-9.4.0-2.fm3_10.el8.noarch rubygem-hammer_cli_foreman-3.10.0-1.el8.noarch rubygem-puppetdb_foreman-6.0.2-1.fm3_9.el8.noarch
Distribution and version
Rocky 8.9