blackducksoftware / perceivers

Kubernetes and openshift image discovery for OpsSight
Apache License 2.0
4 stars 7 forks source link

pod perceiver crashes when perceptor is down #24

Closed mattfenwick closed 6 years ago

mattfenwick commented 6 years ago

steps:

log:


time="2018-03-02T16:32:14Z" level=info msg="starting pod-perceiver"
--
  | time="2018-03-02T16:32:14Z" level=info msg="starting pod controllers"
  | time="2018-03-02T16:32:14Z" level=info msg="starting pod dumper controller"
  | time="2018-03-02T16:32:14Z" level=info msg="starting pod controller"
  | time="2018-03-02T16:32:14Z" level=info msg="starting pod annotator controller"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod default/things-b7kz3"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod hub/postgres-gs4pf"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod default/hi-4qrkz"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod default/registry-console-1-s7j1n"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod bds-perceptor/protoform"
  | E0302 16:32:14.963513       1 pod_controller.go:165] bds-perceptor/protoform failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod bds-perceptor/prometheus-627716970-wmkzp"
  | E0302 16:32:14.963502       1 pod_controller.go:165] default/registry-console-1-s7j1n failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:14.963558       1 pod_controller.go:165] default/hi-4qrkz failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:14.963595       1 pod_controller.go:165] default/things-b7kz3 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:14.963647       1 pod_controller.go:165] hub/postgres-gs4pf failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod hub/documentation-tfrtn"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod default/lotsandlots-p9gvr"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod hub/nginx-webapp-logstash-4qx35"
  | time="2018-03-02T16:32:14Z" level=info msg="processing pod hub/registration-hq8b9"
  | E0302 16:32:17.969478       1 pod_controller.go:165] hub/documentation-tfrtn failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:17.969493       1 pod_controller.go:165] bds-perceptor/prometheus-627716970-wmkzp failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:17.969517       1 pod_controller.go:165] hub/registration-hq8b9 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:17Z" level=info msg="processing pod hub/solr-kcnj3"
  | E0302 16:32:17.969567       1 pod_controller.go:165] default/lotsandlots-p9gvr failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:17.969569       1 pod_controller.go:165] hub/nginx-webapp-logstash-4qx35 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:17Z" level=info msg="processing pod default/blahblah-nbkbx"
  | time="2018-03-02T16:32:17Z" level=info msg="processing pod default/sti-ruby-1-zmcnk"
  | time="2018-03-02T16:32:17Z" level=info msg="processing pod hub/hub-scan-4hwfd"
  | time="2018-03-02T16:32:17Z" level=info msg="processing pod bds-perceptor/perceptor-scanner-t7wkd"
  | E0302 16:32:22.979390       1 pod_controller.go:165] hub/solr-kcnj3 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:22.979450       1 pod_controller.go:165] hub/hub-scan-4hwfd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:22.979390       1 pod_controller.go:165] bds-perceptor/perceptor-scanner-t7wkd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:22.979391       1 pod_controller.go:165] default/sti-ruby-1-zmcnk failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:22Z" level=info msg="processing pod default/sti-ruby-1-build"
  | time="2018-03-02T16:32:22Z" level=info msg="processing pod hub/zookeeper-t120l"
  | time="2018-03-02T16:32:22Z" level=info msg="processing pod bds-perceptor/pod-perceiver-f60gz"
  | time="2018-03-02T16:32:22Z" level=info msg="processing pod hub/jobrunner-fmx3l"
  | E0302 16:32:25.985510       1 pod_controller.go:165] default/blahblah-nbkbx failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:25.985555       1 pod_controller.go:165] hub/jobrunner-fmx3l failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:25.985510       1 pod_controller.go:165] bds-perceptor/pod-perceiver-f60gz failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:25Z" level=info msg="processing pod default/rookie-5k9f0"
  | E0302 16:32:25.985576       1 pod_controller.go:165] default/sti-ruby-1-build failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:25.985510       1 pod_controller.go:165] hub/zookeeper-t120l failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:25Z" level=info msg="processing pod hub/cfssl-fhwzg"
  | time="2018-03-02T16:32:25Z" level=info msg="processing pod bds-perceptor/perceptor-scanner-9vf30"
  | time="2018-03-02T16:32:25Z" level=info msg="processing pod default/router-1-gxz3t"
  | time="2018-03-02T16:32:25Z" level=info msg="processing pod default/docker-registry-1-fw9qd"
  | E0302 16:32:28.991682       1 pod_controller.go:165] default/router-1-gxz3t failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:28Z" level=info msg="processing pod bds-perceptor/protoform"
  | E0302 16:32:28.991753       1 pod_controller.go:165] default/docker-registry-1-fw9qd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:28.991778       1 pod_controller.go:165] bds-perceptor/perceptor-scanner-9vf30 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:28.991682       1 pod_controller.go:165] default/rookie-5k9f0 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:28.991809       1 pod_controller.go:165] hub/cfssl-fhwzg failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:28Z" level=info msg="processing pod default/registry-console-1-s7j1n"
  | time="2018-03-02T16:32:28Z" level=info msg="processing pod default/hi-4qrkz"
  | time="2018-03-02T16:32:28Z" level=info msg="processing pod default/things-b7kz3"
  | time="2018-03-02T16:32:28Z" level=info msg="processing pod hub/postgres-gs4pf"
  | E0302 16:32:31.997472       1 pod_controller.go:165] default/registry-console-1-s7j1n failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:31.997529       1 pod_controller.go:165] hub/postgres-gs4pf failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:31Z" level=info msg="processing pod hub/documentation-tfrtn"
  | E0302 16:32:31.997564       1 pod_controller.go:165] default/things-b7kz3 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:31.997472       1 pod_controller.go:165] bds-perceptor/protoform failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:31.997582       1 pod_controller.go:165] default/hi-4qrkz failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:31Z" level=info msg="processing pod bds-perceptor/prometheus-627716970-wmkzp"
  | time="2018-03-02T16:32:31Z" level=info msg="processing pod hub/registration-hq8b9"
  | time="2018-03-02T16:32:32Z" level=info msg="processing pod default/lotsandlots-p9gvr"
  | time="2018-03-02T16:32:32Z" level=info msg="processing pod hub/nginx-webapp-logstash-4qx35"
  | E0302 16:32:35.003706       1 pod_controller.go:165] bds-perceptor/prometheus-627716970-wmkzp failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:35.003755       1 pod_controller.go:165] hub/registration-hq8b9 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:35.003766       1 pod_controller.go:165] hub/nginx-webapp-logstash-4qx35 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:35Z" level=info msg="processing pod hub/solr-kcnj3"
  | E0302 16:32:35.003783       1 pod_controller.go:165] hub/documentation-tfrtn failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:35.003707       1 pod_controller.go:165] default/lotsandlots-p9gvr failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:35Z" level=info msg="processing pod hub/hub-scan-4hwfd"
  | time="2018-03-02T16:32:35Z" level=info msg="processing pod bds-perceptor/perceptor-scanner-t7wkd"
  | time="2018-03-02T16:32:35Z" level=info msg="processing pod default/sti-ruby-1-zmcnk"
  | time="2018-03-02T16:32:35Z" level=info msg="processing pod default/blahblah-nbkbx"
  | E0302 16:32:38.009566       1 pod_controller.go:165] hub/hub-scan-4hwfd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:38.009614       1 pod_controller.go:165] default/sti-ruby-1-zmcnk failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:38Z" level=info msg="processing pod hub/jobrunner-fmx3l"
  | E0302 16:32:38.009637       1 pod_controller.go:165] default/blahblah-nbkbx failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:38.009566       1 pod_controller.go:165] hub/solr-kcnj3 failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:38.009664       1 pod_controller.go:165] bds-perceptor/perceptor-scanner-t7wkd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:38Z" level=info msg="processing pod bds-perceptor/pod-perceiver-f60gz"
  | time="2018-03-02T16:32:38Z" level=info msg="processing pod default/sti-ruby-1-build"
  | time="2018-03-02T16:32:38Z" level=info msg="processing pod hub/zookeeper-t120l"
  | time="2018-03-02T16:32:38Z" level=info msg="processing pod default/router-1-gxz3t"
  | E0302 16:32:43.019344       1 pod_controller.go:165] bds-perceptor/pod-perceiver-f60gz failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:43.019392       1 pod_controller.go:165] hub/jobrunner-fmx3l failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:43Z" level=info msg="processing pod default/docker-registry-1-fw9qd"
  | E0302 16:32:43.019441       1 pod_controller.go:165] default/sti-ruby-1-build failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:43.019350       1 pod_controller.go:165] default/router-1-gxz3t failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | E0302 16:32:43.019515       1 pod_controller.go:165] hub/zookeeper-t120l failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:43Z" level=info msg="processing pod bds-perceptor/perceptor-scanner-9vf30"
  | time="2018-03-02T16:32:43Z" level=info msg="processing pod default/rookie-5k9f0"
  | time="2018-03-02T16:32:43Z" level=info msg="processing pod hub/cfssl-fhwzg"
  | time="2018-03-02T16:32:43Z" level=info msg="processing pod default/registry-console-1-s7j1n"
  | time="2018-03-02T16:32:44Z" level=info msg="attempting to GET http://perceptor:3001/scanresults for pod annotation"
  | E0302 16:32:46.025716       1 pod_controller.go:165] default/docker-registry-1-fw9qd failed with : unable to POST to http://perceptor:3001/pod: Post http://perceptor:3001/pod: dial tcp 172.30.76.160:3001: connect: no route to host
  | time="2018-03-02T16:32:46Z" level=info msg="GET to http://perceptor:3001/scanresults succeeded, about to update annotations on all pods"
  | panic: runtime error: invalid memory address or nil pointer dereference
  | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x456b5a]
  | goroutine 22 [running]:
  | github.com/blackducksoftware/perceivers/pod/pkg/annotator.(*PodAnnotator).Run(0xc420328480, 0x6fc23ac00, 0xc4202a6420)
  | /workspace/gopath/src/github.com/blackducksoftware/perceivers/pod/pkg/annotator/pod_annotator.go:82 +0x22c
  | created by github.com/blackducksoftware/perceivers/pod/cmd/app.(*PodPerceiver).Run
  | /workspace/gopath/src/github.com/blackducksoftware/perceivers/pod/cmd/app/podperceiver.go:77 +0xbc