Sentry::Cron::MonitorCheckIns not creating check-ins automatically

trevorturk commented 1 year ago

Issue Description

Via https://github.com/getsentry/sentry-ruby/pull/2130

Relevant project ID is 294023, sentry-rails (5.12.0) with the following in a job:

class Api::RedisHitCounterDbSyncJob
  include Sentry::Cron::MonitorCheckIns
  sentry_monitor_check_ins monitor_config: Sentry::Cron::MonitorConfig.from_crontab("0 * * * * *")

I'm seeing the blank slate view on the sentry.io/crons page, but expect to see check-ins start appearing.

Reproduction Steps

Without configuration in the web admin, without any cross set up already, add the minimal required Sentry::Cron::MonitorCheckIns with config.

Expected Behavior

Expect to see check-ins on sentry.io/crons

Actual Behavior

See the blank slate on sentry.io/crons

Ruby Version

3.2.2

SDK Version

sentry-rails (5.11.0)

Integration and Its Version

Sidekiq 6.5.11

Sentry Config

require "active_support/parameter_filter"

Sentry.init do |config|
  config.excluded_exceptions = Helloweather::Application.config.ignored_exceptions
  config.enabled_environments = ["production"]
  config.send_default_pii = true
  filter = ActiveSupport::ParameterFilter.new(Rails.application.config.filter_parameters)
  config.before_send = lambda do |event, hint|
    event.request.url.gsub!(/[a-zA-Z0-9]{32}/, "[FILTERED]")
    filter.filter(event.to_hash)
  end
end

sl0thentr0py commented 1 year ago

is class Api::RedisHitCounterDbSyncJob an active job or sidekiq job? We patch the perform method so if it's some custom job class our module won't work.

trevorturk commented 1 year ago

This is more code than you need, but the full class looks like this:

class Api::RedisHitCounterDbSyncJob
  include Sidekiq::Job
  include Sentry::Cron::MonitorCheckIns

  sidekiq_options retry: false
  sentry_monitor_check_ins monitor_config: Sentry::Cron::MonitorConfig.from_crontab("0 * * * * *")

  def perform(*args)
    today = Time.now.utc.to_date
    # sync today's data, and ensure we've sync'd all data from yesterday
    [today, today.yesterday].each do |date|
      Api::RedisHitCounter.db_sync(date: date)
    end
  end
end

It's being run (and I verified it's actually running) via Sidekiq Scheduler:

:scheduler:
  :schedule:
    redis_hit_counter_db_sync_job:
      cron: "0 * * * * *" # 1 minute interval, should match sentry_monitor_check_ins
      class: Api::RedisHitCounterDbSyncJob

...not sure if perhaps the include Sidekiq::Job doesn't do the trick for some reason? (Note also I'm on an older Sidekiq because Redis.com hasn't enabled RESP3 for me yet.)

sl0thentr0py commented 1 year ago

hey @trevorturk so I checked again and this should work. We actually had an ingestion incident when you were trying this out I think which is very likely why you didn't see the data reflected in the product UI. :(

Can you try out once again? Sorry for the inconvenience!

trevorturk commented 1 year ago

I'm not seeing anything new on sentry.io/crons, and I left the code changes in place. Would I have to redeploy to try to kick things into gear again?

sl0thentr0py commented 1 year ago

hmm alright I think I found the issue, we don't support 6 field crontab so ingest drops the event. Can you make it a 5 field crontab , so just * * * * * if this is every minute. Sorry we're still ironing out some parts of the system, I'll pass on feedback about this too.

trevorturk commented 12 months ago

Hmm I just deployed the following, but I'm not seeing anything on the /crons page

sentry_monitor_check_ins monitor_config: Sentry::Cron::MonitorConfig.from_interval(1, :minute)

sl0thentr0py commented 12 months ago

hmm that should really work, I'm all out of ideas now, here's a sanity checklist

make sure SENTRY_DSN is available to the sidekiq workers environment since you're not setting it in the init call

check that RedisHitCounterDbSyncJob.ancestors has the sentry patches

RedisHitCounterDbSyncJob.ancestors
=> [Sentry::Cron::MonitorCheckIns::Patch, RedisHitCounterDbSyncJob, Sentry::Cron::MonitorCheckIns, ...]

if nothing still works, manually trigger a RedisHitCounterDbSyncJob.perform_async and see if it sends everything

turn on debug logging with

config.debug = true
config.logger.level = ::Logger::DEBUG

it should post stuff like

I, [2023-11-15T13:56:34.917862 #67991]  INFO -- sentry: [Transport] Sending envelope with items [check_in] cde30047f53b446baff978142f8225a6 to Sentry

We will also add some statistics server side for invalid json payloads but they're still not live so I can't tell you right now if the server is dropping stuff for some reason.

trevorturk commented 11 months ago

Ah, thank you for the debugging tips, I was able to find the issue. I'd been doing some manual filtering like so:

  config.before_send = lambda do |event, hint|
    event.request.url.gsub!(/[a-zA-Z0-9]{32}/, "[FILTERED]")
    filter.filter(event.to_hash)
  end

Adjusted to:

event.request&.url&.gsub!(/[a-zA-Z0-9]{32}/, "[FILTERED]")

Which solves the issue. I'm sorry for missing that, I hadn't realized anything was wrong with the config for background jobs! I think this is fair to close, but might be worth suggesting these debugging tips (looking at logs, really) in the docs?

sl0thentr0py commented 11 months ago

i'll add some troubleshooting section to the docs, glad it worked!

getsentry / sentry-ruby