search

voyagermesh / voyager

🚀 Secure L7/L4 (HAProxy) Ingress Controller for Kubernetes
https://voyagermesh.com
Apache License 2.0
1.35k stars 134 forks source link

Failed to list ServiceMonitor #847

Closed serathius closed 6 years ago

serathius commented 6 years ago

Kubernetes Provider: GKE Kubernetes Version: v1.8.6-gke.0 Voyager Deployment Script: curl -fsSL https://raw.githubusercontent.com/appscode/voyager/5.0.0-rc.11/hack/deploy/voyager.sh | bash -s -- --provider=gke --rbac
Prometheus Operator Version: v0.16.0

Voyager Logs: 

I0208 09:10:41.062185       1 logs.go:19] FLAG: --address=":56790"
I0208 09:10:41.062409       1 logs.go:19] FLAG: --alsologtostderr="false"
I0208 09:10:41.062491       1 logs.go:19] FLAG: --analytics="true"
I0208 09:10:41.062583       1 logs.go:19] FLAG: --burst="1000000"
I0208 09:10:41.062669       1 logs.go:19] FLAG: --cloud-config=""
I0208 09:10:41.062751       1 logs.go:19] FLAG: --cloud-provider="gke"
I0208 09:10:41.062832       1 logs.go:19] FLAG: --custom-templates=""
I0208 09:10:41.062920       1 logs.go:19] FLAG: --exporter-sidecar-image="appscode/voyager:5.0.0-rc.11"
I0208 09:10:41.063004       1 logs.go:19] FLAG: --haproxy-image="appscode/haproxy:1.7.9-5.0.0-rc.11"
I0208 09:10:41.063088       1 logs.go:19] FLAG: --haproxy.server-metric-fields="2,3,4,5,6,7,8,9,13,14,15,16,17,18,21,24,33,35,38,39,40,41,42,43,44"
I0208 09:10:41.063180       1 logs.go:19] FLAG: --haproxy.timeout="5s"
I0208 09:10:41.063260       1 logs.go:19] FLAG: --help="false"
I0208 09:10:41.063347       1 logs.go:19] FLAG: --ingress-class="voyager"
I0208 09:10:41.063440       1 logs.go:19] FLAG: --kubeconfig=""
I0208 09:10:41.063532       1 logs.go:19] FLAG: --log.format="\"logger:stderr\""
I0208 09:10:41.063615       1 logs.go:19] FLAG: --log.level="\"info\""
I0208 09:10:41.063695       1 logs.go:19] FLAG: --log_backtrace_at=":0"
I0208 09:10:41.063773       1 logs.go:19] FLAG: --log_dir=""
I0208 09:10:41.064340       1 logs.go:19] FLAG: --logtostderr="true"
I0208 09:10:41.064517       1 logs.go:19] FLAG: --master=""
I0208 09:10:41.064604       1 logs.go:19] FLAG: --operator-service="voyager-operator"
I0208 09:10:41.064692       1 logs.go:19] FLAG: --qps="1e+06"
I0208 09:10:41.064777       1 logs.go:19] FLAG: --rbac="true"
I0208 09:10:41.064847       1 logs.go:19] FLAG: --restrict-to-operator-namespace="false"
I0208 09:10:41.064924       1 logs.go:19] FLAG: --resync-period="5m0s"
I0208 09:10:41.064996       1 logs.go:19] FLAG: --stderrthreshold="2"
I0208 09:10:41.065073       1 logs.go:19] FLAG: --v="3"
I0208 09:10:41.065143       1 logs.go:19] FLAG: --vmodule=""
W0208 09:10:41.277092       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0208 09:10:41.278596       1 run.go:121] Starting Voyager operator...
I0208 09:10:41.278680       1 operator.go:55] Ensuring CRD registration
I0208 09:10:44.379682       1 validator.go:26] Skipping ingress external@default, as it is not handled by Voyager.
I0208 09:10:44.481711       1 run.go:153] URL pattern: /:apiGroup/v1beta1/namespaces/:namespace/ingresses/:name/metrics
I0208 09:10:44.481906       1 run.go:157] Listening on :56790
I0208 09:10:44.556550       1 reflector.go:202] Starting reflector *v1.Service (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556595       1 reflector.go:240] Listing and watching *v1.Service from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556550       1 reflector.go:202] Starting reflector *v1.Endpoints (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556929       1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557119       1 reflector.go:202] Starting reflector *v1.Node (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557133       1 reflector.go:240] Listing and watching *v1.Node from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557432       1 reflector.go:202] Starting reflector *v1.ConfigMap (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557453       1 reflector.go:240] Listing and watching *v1.ConfigMap from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.556983       1 reflector.go:202] Starting reflector *v1.Namespace (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557873       1 reflector.go:240] Listing and watching *v1.Namespace from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558105       1 reflector.go:202] Starting reflector *v1.Secret (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558122       1 reflector.go:240] Listing and watching *v1.Secret from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558318       1 reflector.go:202] Starting reflector *v1beta1.Ingress (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558419       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558638       1 reflector.go:202] Starting reflector *v1beta1.Ingress (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558658       1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558872       1 reflector.go:202] Starting reflector *v1beta1.Certificate (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.558889       1 reflector.go:240] Listing and watching *v1beta1.Certificate from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559091       1 reflector.go:202] Starting reflector *v1.ServiceMonitor (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559113       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557802       1 reflector.go:202] Starting reflector *v1beta1.DaemonSet (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559193       1 reflector.go:240] Listing and watching *v1beta1.DaemonSet from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.557945       1 reflector.go:202] Starting reflector *v1beta1.Deployment (5m0s) from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.559321       1 reflector.go:240] Listing and watching *v1beta1.Deployment from github.com/appscode/voyager/pkg/operator/operator.go:84
I0208 09:10:44.594146       1 services.go:38] [d160fb60-5f1a-46eb-9b69-53568febd56c] Service guestbook@default added
E0208 09:10:44.608339       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
I0208 09:10:44.619965       1 ingresses.go:35] [0a50a1c9-517f-417f-8bf3-c602399abdff] /, Kind= external@default added
I0208 09:10:44.620188       1 ingresses.go:43] [0a50a1c9-517f-417f-8bf3-c602399abdff] /, Kind= external@default does not match ingress class
I0208 09:10:44.721969       1 services.go:38] [e634eb3b-adab-4e5e-8259-c71f05c96a55] Service kubernetes@default added
I0208 09:10:44.819468       1 services.go:38] [0d729567-ca42-4c2d-8a31-fb30257ea700] Service redis-master@default added
I0208 09:10:44.841952       1 services.go:38] [54a18968-1391-410b-933d-cdaffb137bb4] Service redis-slave@default added
I0208 09:10:44.854698       1 services.go:38] [1c02d01b-5529-40f6-b13c-5c2942aab8d0] Service voyager-operator@default added
I0208 09:10:44.862423       1 services.go:38] [88867376-a353-4692-b14f-fca4bc6b6256] Service default-http-backend@ingress-nginx added
I0208 09:10:44.872821       1 services.go:38] [e26422a4-bc54-4a9e-bc46-169801fd718f] Service ingress-metrics@ingress-nginx added
I0208 09:10:44.881610       1 services.go:38] [19000b50-e5f7-41ee-91f4-289a656b6e3d] Service ingress-nginx@ingress-nginx added
I0208 09:10:44.890661       1 services.go:38] [e41d7936-3ac7-42c1-a32d-1d2a43d6cd81] Service default-http-backend@kube-system added
I0208 09:10:44.898731       1 services.go:38] [4d3c42be-cd0d-461f-8380-3c931068e2a2] Service heapster@kube-system added
I0208 09:10:44.906057       1 services.go:38] [ac006cc5-b8d5-48a4-9916-bb8754ff0c79] Service kube-dns@kube-system added
I0208 09:10:44.913592       1 services.go:38] [95770477-94c1-4c37-bf11-a65f1d2946b2] Service kubelet@kube-system added
I0208 09:10:44.921449       1 services.go:38] [6548fb58-e4eb-417a-a017-ebb63f1b5307] Service kubernetes-dashboard@kube-system added
I0208 09:10:44.928805       1 services.go:38] [997409c6-d407-4ee9-9d63-faf8e824d0fe] Service alertmanager-main@monitoring added
I0208 09:10:44.938681       1 services.go:38] [de036c50-eaa5-4777-80eb-d4b9747364eb] Service alertmanager-operated@monitoring added
I0208 09:10:44.946436       1 services.go:38] [5aca2abd-7cd6-4eda-9adf-349892429139] Service grafana@monitoring added
I0208 09:10:44.954349       1 services.go:38] [b0ae2ca1-eb2b-4b3e-830b-69ff1752b366] Service kube-state-metrics@monitoring added
I0208 09:10:44.973430       1 services.go:38] [0fd7f082-49c4-4783-af8e-d1945f2ba510] Service node-exporter@monitoring added
I0208 09:10:44.988000       1 services.go:38] [caf6367d-7ba2-41b2-840c-071facc2c929] Service prometheus-k8s@monitoring added
I0208 09:10:45.007686       1 services.go:38] [7b614b2a-4e7d-4ded-943f-b6695775da46] Service prometheus-operated@monitoring added
I0208 09:10:45.023737       1 services.go:38] [2e8525ab-cc52-42ff-87a7-e8369ccf053f] Service prometheus-operator@monitoring added
I0208 09:10:45.037730       1 services.go:38] [f9ecef03-cd22-497d-9272-64e86fd98743] Service voyager-operator@monitoring added
I0208 09:10:46.047431       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
E0208 09:10:46.094608       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
I0208 09:10:47.228669       1 reflector.go:240] Listing and watching *v1.ServiceMonitor from github.com/appscode/voyager/pkg/operator/operator.go:84
E0208 09:10:47.261644       1 reflector.go:205] github.com/appscode/voyager/pkg/operator/operator.go:84: Failed to list *v1.ServiceMonitor: unknown (get servicemonitors.monitoring.coreos.com)
...

Operator gets stuck on trying to list service monitors.

tamalsaha commented 6 years ago

@serathius , operator will not register watcher for ServiceMonitor if you don't have ServiceMonitor CRD registered in your cluster. https://github.com/appscode/voyager/blob/release-5.0/pkg/operator/servicemonitors.go#L20

You mentioned you are using coreos operator 0.16.x. So, I suppose you have this CRD. So, the issue is we can missing RBAC in the installer https://github.com/appscode/voyager/blob/release-5.0/hack/deploy/rbac.yaml#L29 . Please change it to 

- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs: ["get", "create", "update", "patch", "list", "watch"]

This should fix it.

tamalsaha commented 6 years ago

Thanks for reporting the issue !