search

getlago / lago-api

Open Source Metering and Usage Based Billing
https://www.getlago.com/
GNU Affero General Public License v3.0
331 stars 98 forks source link

[BUG]: db:migrate:primary failing on RunCustomerCurrencyTask when deploying helm release #1864

Closed jeff1010322 closed 5 months ago

jeff1010322 commented 5 months ago

Describe the bug Trying to test out Lago by deploying using the helm chart: https://github.com/getlago/lago-helm-charts Using PostgreSQL 14 on Google Cloud for the DB. The migrate-job always fails at the RunCustomerCurrencyTask.

The previous migration jobs worked and created tables. But this and all later migrations will not run due to it failing each time on this migration. Looking for any information regarding what may be causing this particular migration to fail.

To Reproduce Steps to reproduce the behavior:

  1. Deploy helm chart https://github.com/getlago/lago-helm-charts with databaseUrl set.
  2. Watch migrate-job

As a note I did fork and run my own version of the helm chart mainly just to add in trace flags and manually removed the postgres dependency as I was unable to prevent it from deploying postgres containers with just the configs.

The migrate-job appears to be running this command:

bundle exec rake db:migrate:primary

Expected behavior Migrate job to finish successfully and initialize the DB.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context lago-migrate logs:

** Invoke db:migrate:primary (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Execute db:migrate:primary
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled: (StandardError)

Connection reset by peer
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:101:in `rescue in read'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:92:in `read'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:51:in `block in call_pipelined'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:49:in `times'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:49:in `call_pipelined'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:713:in `block in connect'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:712:in `connect'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:674:in `raw_connection'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:641:in `ensure_connected'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:376:in `pipelined'
/usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/decorator.rb:51:in `pipelined'
/usr/local/bundle/gems/sidekiq-7.1.4/lib/sidekiq/client.rb:226:in `block in raw_push'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
/usr/local/bundle/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
/usr/local/bundle/gems/sidekiq-7.1.4/lib/sidekiq/client.rb:223:in `raw_push'
/usr/local/bundle/gems/sidekiq-7.1.4/lib/sidekiq/client.rb:93:in `push'
/usr/local/bundle/gems/sidekiq-7.1.4/lib/sidekiq/client.rb:172:in `push'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/queue_adapters/sidekiq_adapter.rb:30:in `enqueue_at'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:65:in `block in enqueue'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/active_job.rb:76:in `block in run_in_trace'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/method_tracer.rb:74:in `block in trace_execution_scoped'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/method_tracer_helpers.rb:37:in `block in trace_execution_scoped'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/method_tracer_helpers.rb:37:in `trace_execution_scoped'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/method_tracer.rb:72:in `trace_execution_scoped'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/active_job.rb:74:in `run_in_trace'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/active_job.rb:51:in `enqueue'
/usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/active_job.rb:19:in `block (4 levels) in <main>'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:53:in `block (2 levels) in <module:ActiveJobPatch>'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
/usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
/usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/configured_job.rb:15:in `perform_later'
/app/db/migrate/20220919133338_run_customer_currency_task.rb:6:in `change'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:905:in `exec_migration'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:889:in `block (2 levels) in migrate'
/usr/local/lib/ruby/3.2.0/benchmark.rb:296:in `measure'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:888:in `block in migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:887:in `migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1095:in `migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1409:in `block in execute_migration_in_transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/transactions.rb:209:in `transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1460:in `ddl_transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/usr/local/bundle/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:145:in `block (4 levels) in <main>'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:281:in `block in execute'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:281:in `each'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:281:in `execute'
/usr/local/bundle/gems/sentry-ruby-5.12.0/lib/sentry/rake.rb:24:in `execute'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:199:in `synchronize'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/task.rb:188:in `invoke'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:182:in `invoke_task'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `each'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:138:in `block in top_level'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:147:in `run_with_threads'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:132:in `top_level'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:83:in `block in run'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:208:in `standard_exception_handling'
/usr/local/bundle/gems/rake-13.1.0/lib/rake/application.rb:80:in `run'
/usr/local/bundle/gems/rake-13.1.0/exe/rake:27:in `<top (required)>'
/usr/local/bundle/bin/rake:25:in `load'
/usr/local/bundle/bin/rake:25:in `<top (required)>'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `load'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:23:in `run'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:451:in `exec'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:34:in `dispatch'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:28:in `start'
/usr/local/bundle/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
/usr/local/bundle/gems/bundler-2.5.5/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/local/bundle/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
/usr/local/bundle/bin/bundle:25:in `load'
/usr/local/bundle/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate:primary
== 20220919133338 RunCustomerCurrencyTask: migrating ==========================
{"ts":"2024-04-11T17:30:16.287Z","pid":1,"tid":"4x9","lvl":"INFO","msg":"Sidekiq 7.1.4 connecting to Redis with options {:size=>10, :pool_name=>\"internal\", :url=>\"redis://lago-redis-master.billing:6379\", :pool_timeout=>5}"}
Stream closed EOF for billing/lago-migrate-986zc (lago-migrate)

Version

jeff1010322 commented 5 months ago

For additional info, I also disabled the wait-for-migration init container conditions since I was having this issue just for testing purposes. The frontend and api were able to run after this, but getting this error when trying to do a sign up:

NoMethodError (undefined method `default_currency'

I see that this is likely related to this migration not finishing.

jeff1010322 commented 5 months ago

I figured out what was causing the issue. In my case the redis service was not coming up quick enough and the lago-migrate was failing to make the connection. When I updated the restart policy to OnFailure for the migrate job it worked on the second retry.

Not sure what the official solution for this should be. Maybe an init container check for redis on the migrate job in the helm charts. But either way doesn't appear to be an issue with the lago API.

jdenquin commented 5 months ago

@jeff1010322, ho yes I can add a wait for redis also on migrate job, it is already the case on api !