search

apache / trafficcontrol

Apache Traffic Control is an Open Source implementation of a Content Delivery Network
https://trafficcontrol.apache.org/
Apache License 2.0
1.02k stars 339 forks source link

Traffic Stats shouldn't panic when it cannot talk to Traffic OPs #2692

Open dneuman64 opened 5 years ago

dneuman64 commented 5 years ago

If Traffic Stats cannot talk to Traffic Ops, like to get servers, it panics and dies. Traffic Stats should be able to handle this situation without panicking.

Last login: Thu Jul 26 14:56:45 2018 from 10.169.232.127 panic: Error getting server list from https://trafficops.cdn.net: Get https://trafficops.cdn.net/api/1.2/servers.json: net/http: request canceled

goroutine 9 [running]: panic(0x7b3bc0, 0xc820193060) /usr/lib/golang/src/runtime/panic.go:481 +0x3e6 main.getToData(0xc82000f9e8, 0x6, 0xc82000fa00, 0xe, 0xc8200c0ae0, 0x1e, 0xc82000fa10, 0xa, 0x0, 0x0, ...) /tmp/trafficcontrol/rpmbuild/BUILD/traffic_stats-2.2.0/src/github.com/apache/incubator-trafficcontrol/traffic_stats/traffic_stats.go:447 +0x63b created by main.main /tmp/trafficcontrol/rpmbuild/BUILD/traffic_stats-2.2.0/src/github.com/apache/incubator-trafficcontrol/traffic_stats/traffic_stats.go:129 +0x280 [FAILED]

rawlinp commented 3 years ago

@dneuman64 looking at the code for this, it seems like it's supposed to panic only when it's initializing. Once it's initialized, it looks like it just logs an error and continues working. Is that the desired behavior? Or do we want TS to not panic on startup/initialization if it can't reach TO? Perhaps it should retry on startup instead?