[BUG]: docker containers don't crash

haf commented 1 year ago

Describe the bug If the database, or redis, is unavailable, the docker containers don't crash. This stops them from auto-healing (e.g. DNS recovering and injecting the env var).

To Reproduce Steps to reproduce the behavior:

Deploy e.g. api-worker, the database, but leave out the redis URI
The api-worker will now crash-loop internally and log a lot of output (= $$$) without actually crashing the Kubernetes pod.

api-worker-79b46fdf88-7nrnw api-worker W, [2023-02-10T13:33:44.322529 #7]  WARN -- : /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:398:in `rescue in establish_connection'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:379:in `establish_connection'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:115:in `block in connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:344:in `with_reconnect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:114:in `connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `block in connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `block in connect_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `block in with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/tracer.rb:356:in `capture_segment_error'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:10:in `connect_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:25:in `connect'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:417:in `ensure_connected'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:269:in `block in process'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:356:in `logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:78:in `block in logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:17:in `block in instrument'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:28:in `record_span'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:16:in `instrument'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sentry-ruby-core-5.3.1/lib/sentry/redis.rb:77:in `logging'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:268:in `process'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:161:in `call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:17:in `block in call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:17:in `block in call_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `block in with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/tracer.rb:356:in `capture_segment_error'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:55:in `with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:17:in `call_with_tracing'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/newrelic_rpm-8.15.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:17:in `call'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:269:in `block in send_command'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `synchronize'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `send_command'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/redis-4.7.1/lib/redis/commands/sets.rb:11:in `scard'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/api.rb:867:in `block in size'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq.rb:156:in `block in redis'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:65:in `block (2 levels) in with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `handle_interrupt'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:64:in `block in with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `handle_interrupt'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/connection_pool-2.3.0/lib/connection_pool.rb:61:in `with'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq.rb:153:in `redis'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/api.rb:867:in `size'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:190:in `process_count'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:151:in `random_poll_interval'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:120:in `wait'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/scheduled.rb:102:in `block in start'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/component.rb:8:in `watchdog'
api-worker-79b46fdf88-7nrnw api-worker /usr/local/bundle/gems/sidekiq-6.5.1/lib/sidekiq/component.rb:17:in `block in safe_thread'

Expected behavior If the environment a software service runs in is incorrectly configured, print an error message and exit the process. This lets monitoring software alert to the problem. Otherwise ops has to write app-specific event listeners, eventify the logs (including writing stacktrace parsers for ruby) and deploy listeners for the logs, then write API-integrations with the platform runtime (like kubernetes) to restart the pod.

Support

Version: getlago/api:v0.22.0-beta

vincent-pochet commented 1 year ago

Hi @haf - thank you for your message!

It's definitely something that should be improved. We will investigate and get back to you with a fix as soon as possible.

OscarKolsrud commented 12 months ago

Have we any status on this? Seem to be running into something sort of related where the API suddendly returns 500, and a compose down/up fixes it

jdenquin commented 12 months ago

@OscarKolsrud we have to dig it a bit since this is the way Rails/Sidekiq works today, we may have to make it customizable. On our side, on production, we prefer to have errors instead of pods always restarting, but it's a Rubyist habit I think 😂

gabrielseibel1 commented 11 months ago

Hi, my org also noticed this problem, we ended up having a lot of problems because of the lago clock being frozen on error state for 2 days after a redis disconnect (we noticed that it just stopped logging after the error, although running, but customers weren't charged). It's the second or third time we lose a bill-customers day (we suspect) because of this type of outage (pods in error state in last/first days of months). Do you have updates on this, or any suggestion of a mechanism to restart pods once they enter error state?

haf commented 8 months ago

https://en.wikipedia.org/wiki/Crash-only_software - this is a reference to a very sane way of building software - where you auto-correct faults by restarting at different levels.

jdenquin commented 8 months ago

@gabrielseibel1 if Redis is your problem, restarting the pod will not fix the issue, it will restart since Redis will be available again. What are the errors you face on the worker when it's in error?

haf commented 8 months ago

If the connectivity to redis is a problem, a restart will trigger a retry of the connection

jdenquin commented 8 months ago

each time a job is enqueued or want to run, the connection to redis is retried. Is the health check enough for you guys? If the database or redis is out, health endpoint will return an error

haf commented 8 months ago

If it's done using a liveness probe in k8s it would have solved my problem. If you've solved the bug from the stacktrace in the original post in this thread, you can close the issue. Do note that this stacktrace happens on container start though; not after a while / for networking errors - so in the case of this issue - you'd never have a successful health check.

jdenquin commented 8 months ago

I'm currently working on the liveness probe on our helmchart so this is definitely something we'll release very soon!

doctorpangloss commented 5 months ago

it's been a year. the clock crashes often. how do you guys deal with this?

jdenquin commented 4 months ago

@doctorpangloss we never had any clock crash for our cloud environment. If it crash, it's always because of an unhealthy Redis service.

grthr commented 3 months ago

Here is another log from broken clock with lago v1.4. The pod of the lago-clock did not recover within 2 days. There are no errors in the redis logs (newest log from redis is 2024-07-12T05:23:25.509128941Z)

2024-07-13T03:07:09.001473987Z I, [2024-07-13T03:07:09.001191 #7]  INFO -- : Triggering 'schedule:activate_subscriptions'
2024-07-13T03:07:09.004434441Z I, [2024-07-13T03:07:09.004332 #7]  INFO -- : [ActiveJob] Enqueued Clock::ActivateSubscriptionsJob (Job ID: 747ec5ec-3867-4952-bb75-8d954ddd3d3e) to Sidekiq(clock)
2024-07-13T03:07:09.004451474Z I, [2024-07-13T03:07:09.004374 #7]  INFO -- : Finished 'schedule:activate_subscriptions' duration_ms=3 error=nil
2024-07-13T03:07:09.004459389Z I, [2024-07-13T03:07:09.004417 #7]  INFO -- : Triggering 'schedule:refresh_draft_invoices'
2024-07-13T03:07:09.005629697Z I, [2024-07-13T03:07:09.005529 #7]  INFO -- : [ActiveJob] Enqueued Clock::RefreshDraftInvoicesJob (Job ID: e5080ed5-e1df-49cd-a00d-6dbed2cafd4d) to Sidekiq(clock)
2024-07-13T03:07:09.005680141Z I, [2024-07-13T03:07:09.005584 #7]  INFO -- : Finished 'schedule:refresh_draft_invoices' duration_ms=1 error=nil
2024-07-13T03:10:00.002022998Z I, [2024-07-13T03:10:00.001190 #7]  INFO -- : Triggering 'schedule:bill_customers'
2024-07-13T03:10:00.003415444Z I, [2024-07-13T03:10:00.003268 #7]  INFO -- : [ActiveJob] Enqueued Clock::SubscriptionsBillerJob (Job ID: 34ca702a-2be8-4fc5-9775-99396cb619d5) to Sidekiq(clock)
2024-07-13T03:10:00.003464144Z I, [2024-07-13T03:10:00.003396 #7]  INFO -- : Finished 'schedule:bill_customers' duration_ms=2 error=nil
2024-07-13T03:12:09.001506411Z I, [2024-07-13T03:12:09.001196 #7]  INFO -- : Triggering 'schedule:activate_subscriptions'
2024-07-13T03:12:09.211433834Z E, [2024-07-13T03:12:09.211235 #7] ERROR -- : [ActiveJob] Failed enqueuing Clock::ActivateSubscriptionsJob to Sidekiq(clock): Redis::TimeoutError (Waited 0.1 seconds)
2024-07-13T03:12:09.221068222Z E, [2024-07-13T03:12:09.219873 #7] ERROR -- : Waited 0.1 seconds (Redis::TimeoutError)
2024-07-13T03:12:09.221098369Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer'
2024-07-13T03:12:09.221104220Z <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.221108678Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.221112105Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.221115431Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.221119058Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.221122334Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.221126482Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.221130499Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.221134126Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.221137502Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.221159263Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.221163120Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.221166937Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.221171877Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.221175864Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.221183719Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.221187226Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.221190581Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.221194008Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.221200971Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.221204458Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.221207844Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.221211201Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.221214607Z /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.221222892Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.221226549Z /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.221229875Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.221262186Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.221269359Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.221273137Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.221276623Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.221280339Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.221283746Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.221287192Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.221290919Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.221312510Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.221316728Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.221320385Z <internal:numeric>:237:in `times'
2024-07-13T03:12:09.221323831Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.221327969Z /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.221331786Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.221335433Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.221357564Z /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.221362213Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.221365920Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.221369777Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.221373534Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.221377231Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.221380718Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.221384204Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.221388052Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.221394344Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.221397839Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.221401306Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.221404893Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.221408339Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.221411786Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.221415142Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.221418579Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.221422015Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.221425442Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.221428887Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.221432735Z /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.221436532Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.221465366Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.221474193Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.221477940Z /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.221481667Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.221485604Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.221523044Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.221528594Z /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.221532362Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.221564893Z #<Thread:0x00007f93a8327bc8 /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:426 run> terminated with exception (report_on_exception is true):
2024-07-13T03:12:09.221567518Z /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.221601682Z /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.221608213Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.221613473Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.221618873Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.221624624Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.221633461Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.221639402Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.221647257Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.221651615Z /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.221674437Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.221686410Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.221692241Z /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.221699855Z I, [2024-07-13T03:12:09.221140 #7]  INFO -- : Finished 'schedule:activate_subscriptions' duration_ms=220 error=#<Redis::TimeoutError: Waited 0.1 seconds>
2024-07-13T03:12:09.230473462Z /app/clock.rb:13:in `block in <module:Clockwork>': undefined local variable or method `e' for module Clockwork (NameError)
2024-07-13T03:12:09.230529367Z
2024-07-13T03:12:09.230555395Z     Rails.logger.error(e.message)
2024-07-13T03:12:09.230576966Z                        ^
2024-07-13T03:12:09.230623443Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:154:in `handle_error'
2024-07-13T03:12:09.230655022Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:65:in `rescue in execute'
2024-07-13T03:12:09.230688184Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:62:in `execute'
2024-07-13T03:12:09.230748918Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.230812547Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.230855647Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.230886325Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.230943452Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.230984449Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.231049311Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.231117679Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.231194723Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.231231221Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.231268721Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 0.1 seconds (Redis::TimeoutError)
2024-07-13T03:12:09.231363408Z  from <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.231403794Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.231436616Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.231491719Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.231538185Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.231592478Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.231656918Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.231720717Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.231763988Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.231807509Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.231861712Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.231958883Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.231979983Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.231986115Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.231989280Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.231992356Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.231995041Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.231998257Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.232000832Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.232004729Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.232007715Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.232010470Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.232013055Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.232032120Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.232035176Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.232038052Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.232040647Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.232043322Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.232045847Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.232049412Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.232052889Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.232055464Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.232058019Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.232108453Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.232126668Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.232131767Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.232135524Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.232139691Z  from <internal:numeric>:237:in `times'
2024-07-13T03:12:09.232143198Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.232146534Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.232150111Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.232153758Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.232158868Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.232162405Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.232166222Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.232169638Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.232173284Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.232176951Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.232180558Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.232184175Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.232187652Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.232191068Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.232194545Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.232210294Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.232213961Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.232217297Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.232220864Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.232224421Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.232227927Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.232232405Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.232264515Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.232268864Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.232273312Z  from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.232277009Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.232280496Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.232308759Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.232313377Z  from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.232317344Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.232338114Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.232348653Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.232393908Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.232403907Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.232407523Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.232411721Z  from /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.232436327Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.232460493Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.232497852Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.232504134Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.232507631Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.232511538Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.232534751Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.232569587Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.232574767Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.232598881Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.232621293Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'
2024-07-13T03:12:09.232627715Z /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:140:in `block in fill_buffer': Waited 0.1 seconds (RedisClient::ReadTimeoutError)
2024-07-13T03:12:09.232631533Z  from <internal:kernel>:187:in `loop'
2024-07-13T03:12:09.232635500Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:122:in `fill_buffer'
2024-07-13T03:12:09.232660216Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:114:in `ensure_remaining'
2024-07-13T03:12:09.232682678Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/buffered_io.rb:85:in `getbyte'
2024-07-13T03:12:09.232720409Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:113:in `parse'
2024-07-13T03:12:09.232754102Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection/resp3.rb:50:in `load'
2024-07-13T03:12:09.232815587Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/ruby_connection.rb:94:in `read'
2024-07-13T03:12:09.232882703Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/connection_mixin.rb:31:in `call'
2024-07-13T03:12:09.232943798Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:239:in `block (2 levels) in call_v'
2024-07-13T03:12:09.232992048Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client/middlewares.rb:16:in `call'
2024-07-13T03:12:09.233035799Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:238:in `block in call_v'
2024-07-13T03:12:09.233079341Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:643:in `ensure_connected'
2024-07-13T03:12:09.233120158Z  from /usr/local/bundle/gems/redis-client-0.17.0/lib/redis_client.rb:237:in `call_v'
2024-07-13T03:12:09.233161525Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
2024-07-13T03:12:09.233202752Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `block in call_v'
2024-07-13T03:12:09.233244220Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `block in call_with_tracing'
2024-07-13T03:12:09.233284986Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `block in with_tracing'
2024-07-13T03:12:09.233324891Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.233365536Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:59:in `with_tracing'
2024-07-13T03:12:09.233390855Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/instrumentation.rb:19:in `call_with_tracing'
2024-07-13T03:12:09.233446809Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/instrumentation/redis/prepend.rb:12:in `call_v'
2024-07-13T03:12:09.233487956Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
2024-07-13T03:12:09.233542889Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
2024-07-13T03:12:09.233600577Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
2024-07-13T03:12:09.233647365Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:110:in `_eval'
2024-07-13T03:12:09.233674515Z  from /usr/local/bundle/gems/redis-5.0.6/lib/redis/commands/scripting.rb:97:in `evalsha'
2024-07-13T03:12:09.233731071Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'
2024-07-13T03:12:09.233771637Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:152:in `with'
2024-07-13T03:12:09.233812134Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block in lock'
2024-07-13T03:12:09.233852428Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:208:in `recover_from_script_flush'
2024-07-13T03:12:09.233892163Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:170:in `lock'
2024-07-13T03:12:09.233931808Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block (2 levels) in lock_instances'
2024-07-13T03:12:09.233956373Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `select'
2024-07-13T03:12:09.234014953Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:258:in `block in lock_instances'
2024-07-13T03:12:09.234055739Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:311:in `timed'
2024-07-13T03:12:09.234096165Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:257:in `lock_instances'
2024-07-13T03:12:09.234138164Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:232:in `block in try_lock_instances'
2024-07-13T03:12:09.234179010Z  from <internal:numeric>:237:in `times'
2024-07-13T03:12:09.234241717Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:228:in `try_lock_instances'
2024-07-13T03:12:09.234291852Z  from /usr/local/bundle/gems/redlock-1.3.2/lib/redlock/client.rb:69:in `lock'
2024-07-13T03:12:09.234336304Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
2024-07-13T03:12:09.234378263Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
2024-07-13T03:12:09.234419280Z  from /usr/local/bundle/gems/activejob-uniqueness-0.2.5/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
2024-07-13T03:12:09.234459956Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `instance_exec'
2024-07-13T03:12:09.234499390Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:467:in `block in make_lambda'
2024-07-13T03:12:09.234539835Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:199:in `block (2 levels) in halting'
2024-07-13T03:12:09.234580482Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:687:in `block (2 levels) in default_terminator'
2024-07-13T03:12:09.234620116Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `catch'
2024-07-13T03:12:09.234658888Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:686:in `block in default_terminator'
2024-07-13T03:12:09.234698402Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:200:in `block in halting'
2024-07-13T03:12:09.234738137Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `block in invoke_before'
2024-07-13T03:12:09.234777751Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `each'
2024-07-13T03:12:09.234830190Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:595:in `invoke_before'
2024-07-13T03:12:09.234872608Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:116:in `block in run_callbacks'
2024-07-13T03:12:09.234912383Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:25:in `block in instrument'
2024-07-13T03:12:09.234953830Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `block in instrument'
2024-07-13T03:12:09.234994086Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
2024-07-13T03:12:09.235043809Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/notifications.rb:206:in `instrument'
2024-07-13T03:12:09.235086239Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:35:in `instrument'
2024-07-13T03:12:09.235129650Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/instrumentation.rb:9:in `block (2 levels) in <module:Instrumentation>'
2024-07-13T03:12:09.235170626Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.235210852Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.235268951Z  from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `block in tag_logger'
2024-07-13T03:12:09.235299478Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
2024-07-13T03:12:09.235361023Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:37:in `tagged'
2024-07-13T03:12:09.235405837Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/tagged_logging.rb:99:in `tagged'
2024-07-13T03:12:09.235461101Z  from /usr/local/bundle/gems/activejob-traceable-0.4.2/lib/activejob/traceable/logging_patch.rb:17:in `tag_logger'
2024-07-13T03:12:09.235503680Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/logging.rb:14:in `block (2 levels) in <module:Logging>'
2024-07-13T03:12:09.235529719Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `instance_exec'
2024-07-13T03:12:09.235550358Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
2024-07-13T03:12:09.235570195Z  from /usr/local/bundle/gems/activesupport-7.0.8.1/lib/active_support/callbacks.rb:138:in `run_callbacks'
2024-07-13T03:12:09.235589130Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:63:in `enqueue'
2024-07-13T03:12:09.235608587Z  from /usr/local/bundle/gems/activejob-7.0.8.1/lib/active_job/enqueuing.rb:30:in `perform_later'
2024-07-13T03:12:09.235650204Z  from /app/clock.rb:22:in `block in <module:Clockwork>'
2024-07-13T03:12:09.235709686Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:61:in `execute'
2024-07-13T03:12:09.235735484Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/event.rb:48:in `run'
2024-07-13T03:12:09.235791269Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:136:in `block in tick'
2024-07-13T03:12:09.235831033Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `each'
2024-07-13T03:12:09.235871098Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:134:in `tick'
2024-07-13T03:12:09.235911334Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:123:in `block (2 levels) in run_tick_loop'
2024-07-13T03:12:09.235951168Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `synchronize'
2024-07-13T03:12:09.236004968Z  from /usr/local/bundle/gems/clockwork-3.0.2/lib/clockwork/manager.rb:121:in `block in run_tick_loop'
2024-07-13T03:12:09.236048069Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:435:in `block (2 levels) in thread_block_with_current_transaction'
2024-07-13T03:12:09.236088465Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:357:in `capture_segment_error'
2024-07-13T03:12:09.236128750Z  from /usr/local/bundle/gems/newrelic_rpm-9.5.0/lib/new_relic/agent/tracer.rb:434:in `block in thread_block_with_current_transaction'

jdenquin commented 3 months ago

thanks for it @grthr I'm having a look

getlago / lago

[BUG]: docker containers don't crash #188