search

cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data
https://cagov.github.io/caldata-mdsa-caltrans-pems/
MIT License
7 stars 1 forks source link

Relay Server Issue - The Crawl Queue Builds up #237

Closed pingpingxiu-DOT-ca-gov closed 4 months ago

pingpingxiu-DOT-ca-gov commented 5 months ago

Over the last week, the crawl queue starts to build up and exceeded the normal range.

image

This indicates the relaying is malfunctioning.

FYI the dashboard for relay operation is here http://svgcmdl02:5601/app/dashboards#/view/4438c280-fcfd-11ee-99ff-69aa747c5773?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-3w,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!t,title:'Crawling%20Operation',viewMode:view)

Troubleshooting

pingpingxiu-DOT-ca-gov commented 5 months ago

@ZhenyuZhu-Caltrans @ian-r-rose

pingpingxiu-DOT-ca-gov commented 5 months ago

image

A simple restart seems getting the service back to normal.

This needs to be kept watching. If such glitches keep occurring, we may either (1) Set up periodic restart procedure to get around of it, or (2) Deeper look at why it sometimes hanging, needing a restart

This was causing a few days' data gap, which needs backfilling.

@ZhenyuZhu-Caltrans @ian-r-rose

jkarpen commented 5 months ago

@pingpingxiu-DOT-ca-gov will close this issue when the queue is completely finished.

pingpingxiu-DOT-ca-gov commented 4 months ago

image So the backfill crawl queue finished

ian-r-rose commented 4 months ago

Closing, as this incident has been resolved