ScraperMostRecentArchivedFileTimeIsTooOld

[measurementlab]

Alertmanager URL: https://alertmanager.mlab-oti.measurementlab.net

• firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=%28time%28%29+-+%28scraper_maxrawfiletimearchived%7Bcontainer%3D%22scraper-sync%22%7D+%21%3D+0%29%29+%3E+%2856+%2A+60+%2A+60%29+and+ON%28machine%29+%28time%28%29+-+process_start_time_seconds%7Bservice%3D%22sidestream%22%7D%29+%3E+%2830+%2A+60+%2A+60%29+unless+ON%28machine%29+lame_duck_node+%3D%3D+1&g0.tab=1

  Labels:

  • alertname = ScraperMostRecentArchivedFileTimeIsTooOld
  • application = scraper-sync
  • cluster = scraper-cluster-scraper-sync-pool
  • container = scraper-sync
  • deployment = scraper-sync
  • experiment = utility.mlab
  • instance = 10.21.32.7:9090
  • job = kubernetes-pods
  • machine = mlab1.iad01.measurement-lab.org
  • namespace = default
  • pod_template_hash = 3961228517
  • ready = true
  • rsync_module = switch
  • service = prometheus-public-service
  • severity = page
  • zone = us-central1-a

  Annotations:

  • description = Max file mtime for is older than 56 hours.
  • summary = Scraper max file mtime is too old

• firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=%28time%28%29+-+%28scraper_maxrawfiletimearchived%7Bcontainer%3D%22scraper-sync%22%7D+%21%3D+0%29%29+%3E+%2856+%2A+60+%2A+60%29+and+ON%28machine%29+%28time%28%29+-+process_start_time_seconds%7Bservice%3D%22sidestream%22%7D%29+%3E+%2830+%2A+60+%2A+60%29+unless+ON%28machine%29+lame_duck_node+%3D%3D+1&g0.tab=1

  Labels:

  • alertname = ScraperMostRecentArchivedFileTimeIsTooOld
  • application = scraper-sync
  • cluster = scraper-cluster-scraper-sync-pool
  • container = scraper-sync
  • deployment = scraper-sync
  • experiment = utility.mlab
  • instance = 10.21.32.7:9090
  • job = kubernetes-pods
  • machine = mlab1.iad01.measurement-lab.org
  • namespace = default
  • pod_template_hash = 3961228517
  • ready = true
  • rsync_module = utilization
  • service = prometheus-public-service
  • severity = page
  • zone = us-central1-a

  Annotations:

  • description = Max file mtime for is older than 56 hours.
  • summary = Scraper max file mtime is too old

• firing https://prometheus.mlab-oti.measurementlab.net/graph?g0.expr=%28time%28%29+-+%28scraper_maxrawfiletimearchived%7Bcontainer%3D%22scraper-sync%22%7D+%21%3D+0%29%29+%3E+%2856+%2A+60+%2A+60%29+and+ON%28machine%29+%28time%28%29+-+process_start_time_seconds%7Bservice%3D%22sidestream%22%7D%29+%3E+%2830+%2A+60+%2A+60%29+unless+ON%28machine%29+lame_duck_node+%3D%3D+1&g0.tab=1

  Labels:

  • alertname = ScraperMostRecentArchivedFileTimeIsTooOld
  • application = scraper-sync
  • cluster = scraper-cluster-scraper-sync-pool
  • container = scraper-sync
  • deployment = scraper-sync
  • experiment = neubot.mlab
  • instance = 10.21.32.7:9090
  • job = kubernetes-pods
  • machine = mlab1.iad01.measurement-lab.org
  • namespace = default
  • pod_template_hash = 3961228517
  • ready = true
  • rsync_module = neubot
  • service = prometheus-public-service
  • severity = page
  • zone = us-central1-a

  Annotations:

  • description = Max file mtime for is older than 56 hours.
  • summary = Scraper max file mtime is too old

TODO: add graph url from annotations.

[yachang]

resolved by restart iad01 on April 26 night.