Closed measurementlab closed 6 years ago
Appears to be stuck in an rsync list for multiple days. This is different than the timeout bug.
[2018-03-29 08:08:56,817 INFO scraper.py:562 rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute] Removed local file /tmp/20180328T000000Z-mlab1-lga06-paris-traceroute-0000.tgz
[2018-03-29 08:09:00,832 INFO run_scraper.py:237 rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute] Sleeping for 2308.75 seconds
[2018-03-29 08:47:29,673 INFO run_scraper.py:217 rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute] Scraping rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute
[2018-03-29 08:47:29,920 INFO scraper.py:171 rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute] rsync file list discovery from rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute
[2018-03-29 08:47:29,920 INFO scraper.py:224 rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute] Listing files on server with the command: /usr/bin/timeout -s KILL -t 86400 /usr/bin/rsync -n -vv --out-format %n %M -4 -az --bwlimit=10000 --timeout=300 --contimeout=300 --chmod=u=rwX rsync://npad.iupui.mlab1.lga06.measurement-lab.org:7999/paris-traceroute scraper_data/npad.iupui.mlab1.lga06.measurement-lab.org/paris-traceroute
Oh, no -- my mistake that is the timeout bug.
Alertmanager URL: http://status.mlab-oti.measurementlab.net:9093
firing http://status.mlab-oti.measurementlab.net:9090/graph?g0.expr=%28time%28%29+-+%28scraper_maxrawfiletimearchived%7Bcontainer%3D%22scraper-sync%22%7D+%21%3D+0%29%29+%3E+%2856+%2A+60+%2A+60%29+and+ON%28machine%29+%28time%28%29+-+process_start_time_seconds%7Bservice%3D%22sidestream%22%7D%29+%3E+%2830+%2A+60+%2A+60%29+unless+ON%28machine%29+lame_duck_node+%3D%3D+1&g0.tab=1
Labels:
Annotations:
TODO: add graph url from annotations.