search

voyagermesh / voyager

🚀 Secure L7/L4 (HAProxy) Ingress Controller for Kubernetes
https://voyagermesh.com
Apache License 2.0
1.35k stars 134 forks source link

ingress resource not lunching in k8s 1.7 #398

Closed izutheo closed 7 years ago

izutheo commented 7 years ago

Everything works perfectly on our 1.6.x environment but for some reason we have not being able to bring up an ingress resource in k8s 1.7.x which is using weave.

Before we go in any further into our research into this, is there anything that we should be aware of when it comes to creating ingress resource in version 1.7.x. using Appscode?

tamalsaha commented 7 years ago

There are no known issues with Kubernetes 1.7. We are using it with a kops cluster in AWS currently.

Are you using RBAC with 1.7? Then you have to setup voyager appropriately. Are you using Voyager via Helm? Our officially published Voyager chart does not use RBAC yet.

If you can share the voyager operator logs, we will probably have a better idea?

izutheo commented 7 years ago

We are using RBAC with 1.7 and voyager has been setup appropriately taking that into consideration. Here is the log from a quick run that i just performed.

I0816 07:41:05.740111 7 logs.go:19] FLAG: --address=":56790"

I0816 07:41:05.740208 7 logs.go:19] FLAG: --alsologtostderr="false"

I0816 07:41:05.740221 7 logs.go:19] FLAG: --analytics="true"

I0816 07:41:05.740232 7 logs.go:19] FLAG: --cloud-config="$CLOUD_CONFIG"

I0816 07:41:05.740251 7 logs.go:19] FLAG: --cloud-provider="$CLOUD_PROVIDER"

I0816 07:41:05.740259 7 logs.go:19] FLAG: --haproxy-image="appscode/haproxy:1.7.6-3.1.0"

I0816 07:41:05.740269 7 logs.go:19] FLAG: --haproxy.server-metric-fields="2,3,4,5,6,7,8,9,13,14,15,16,17,18,21,24,33,35,38,39,40,41,42,43,44"

I0816 07:41:05.740285 7 logs.go:19] FLAG: --haproxy.timeout="5s"

I0816 07:41:05.740298 7 logs.go:19] FLAG: --help="false"

I0816 07:41:05.740315 7 logs.go:19] FLAG: --http-challenge-port="56791"

I0816 07:41:05.740362 7 logs.go:19] FLAG: --ingress-class="$INGRESS_CLASS"

I0816 07:41:05.740375 7 logs.go:19] FLAG: --kubeconfig=""

I0816 07:41:05.740387 7 logs.go:19] FLAG: --log.format="\"logger:stderr\""

I0816 07:41:05.740396 7 logs.go:19] FLAG: --log.level="\"info\""

I0816 07:41:05.740404 7 logs.go:19] FLAG: --log_backtrace_at=":0"

I0816 07:41:05.740411 7 logs.go:19] FLAG: --log_dir=""

I0816 07:41:05.740418 7 logs.go:19] FLAG: --logtostderr="true"

I0816 07:41:05.740425 7 logs.go:19] FLAG: --master=""

I0816 07:41:05.740432 7 logs.go:19] FLAG: --operator-service="voyager-operator"

I0816 07:41:05.740439 7 logs.go:19] FLAG: --rbac="true"

I0816 07:41:05.740447 7 logs.go:19] FLAG: --stderrthreshold="2"

I0816 07:41:05.740454 7 logs.go:19] FLAG: --v="3"

I0816 07:41:05.740461 7 logs.go:19] FLAG: --vmodule=""

W0816 07:41:35.741317 7 client_config.go:517] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.

I0816 07:41:35.743255 7 run.go:102] Starting Voyager operator...

I0816 07:41:35.743377 7 operator.go:49] Ensuring TPR registration

I0816 07:41:35.787147 7 run.go:116] URL pattern: /:apiGroup/v1beta1/namespaces/:namespace/ingresses/:name/metrics

I0816 07:41:35.787202 7 run.go:120] Listening on :56790

I0816 07:41:35.787579 7 reflector.go:198] Starting reflector *v1.Service (2m0s) from github.com/appscode/voyager/pkg/operator/services.go:56

I0816 07:41:35.787584 7 reflector.go:198] Starting reflector *v1beta1.Deployment (2m0s) from github.com/appscode/voyager/pkg/operator/deployments.go:41

I0816 07:41:35.787671 7 reflector.go:236] Listing and watching *v1.Service from github.com/appscode/voyager/pkg/operator/services.go:56

I0816 07:41:35.787753 7 reflector.go:198] Starting reflector *v1.Namespace (2m0s) from github.com/appscode/voyager/pkg/operator/namespaces.go:46

I0816 07:41:35.787795 7 reflector.go:236] Listing and watching *v1.Namespace from github.com/appscode/voyager/pkg/operator/namespaces.go:46

I0816 07:41:35.788036 7 reflector.go:198] Starting reflector *v1.ConfigMap (2m0s) from github.com/appscode/voyager/pkg/operator/configmaps.go:39

I0816 07:41:35.788081 7 reflector.go:236] Listing and watching *v1.ConfigMap from github.com/appscode/voyager/pkg/operator/configmaps.go:39

I0816 07:41:35.788082 7 reflector.go:198] Starting reflector *v1beta1.DaemonSet (2m0s) from github.com/appscode/voyager/pkg/operator/daemonsets.go:40

I0816 07:41:35.788116 7 reflector.go:236] Listing and watching *v1beta1.DaemonSet from github.com/appscode/voyager/pkg/operator/daemonsets.go:40

I0816 07:41:35.788273 7 reflector.go:198] Starting reflector *api.Ingress (2m0s) from github.com/appscode/voyager/pkg/operator/ingress_tprs.go:81

I0816 07:41:35.788315 7 reflector.go:236] Listing and watching *api.Ingress from github.com/appscode/voyager/pkg/operator/ingress_tprs.go:81

I0816 07:41:35.788395 7 reflector.go:198] Starting reflector *api.Certificate (2m0s) from github.com/appscode/voyager/pkg/operator/certificate_tprs.go:74

I0816 07:41:35.788432 7 reflector.go:236] Listing and watching *api.Certificate from github.com/appscode/voyager/pkg/operator/certificate_tprs.go:74

I0816 07:41:35.788530 7 reflector.go:198] Starting reflector *v1.Endpoints (2m0s) from github.com/appscode/voyager/pkg/operator/endpoints.go:60

I0816 07:41:35.788554 7 reflector.go:236] Listing and watching *v1.Endpoints from github.com/appscode/voyager/pkg/operator/endpoints.go:60

I0816 07:41:35.787673 7 reflector.go:236] Listing and watching *v1beta1.Deployment from github.com/appscode/voyager/pkg/operator/deployments.go:41

I0816 07:41:35.788669 7 reflector.go:198] Starting reflector *v1beta1.Ingress (2m0s) from github.com/appscode/voyager/pkg/operator/ingresses.go:103

I0816 07:41:35.788691 7 reflector.go:236] Listing and watching *v1beta1.Ingress from github.com/appscode/voyager/pkg/operator/ingresses.go:103

I0816 07:41:35.824257 7 services.go:40] Service accountissuingapi@default added

I0816 07:41:35.832038 7 ingress_tprs.go:40] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system added

I0816 07:41:35.832065 7 ingress_tprs.go:42] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system does not match ingress class

I0816 07:41:35.869772 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:41:35.878703 7 endpoints.go:53] Skipping Endpoints kube-controller-manager@kube-system, as it has no matching service

I0816 07:41:35.879718 7 services.go:133] Updating All Ingress, got total 1

I0816 07:41:35.879769 7 services.go:40] Service kubernetes@default added

I0816 07:41:35.893177 7 endpoints.go:53] Skipping Endpoints kube-scheduler@kube-system, as it has no matching service

I0816 07:41:35.909133 7 services.go:133] Updating All Ingress, got total 1

I0816 07:41:35.909201 7 services.go:40] Service kube-dns@kube-system added

I0816 07:41:35.928969 7 services.go:133] Updating All Ingress, got total 1

I0816 07:41:35.929031 7 services.go:40] Service voyager-operator@kube-system added

I0816 07:41:35.949995 7 services.go:133] Updating All Ingress, got total 1

I0816 07:41:36.022855 7 servicemonitors.go:20] Skipping watching non-preferred GroupVersion:monitoring.coreos.com/v1alpha1 Kind:ServiceMonitor

I0816 07:43:35.788564 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:45:21.387920 7 ingress_tprs.go:40] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default added

I0816 07:45:21.387989 7 ingress_tprs.go:42] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default does not match ingress class

I0816 07:45:35.789238 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:45:35.789306 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:47:35.789525 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:47:35.789613 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:49:35.789732 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:49:35.789795 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:51:35.789806 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:51:35.789884 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:53:35.789991 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:53:35.790047 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:55:35.790207 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:55:35.790327 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

I0816 07:57:35.790575 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress testing-ing-wx@kube-system has unchanged spec and annotations

I0816 07:57:35.790640 7 ingress_tprs.go:63] voyager.appscode.com/v1beta1, Kind=Ingress haproxy@default has unchanged spec and annotations

tamalsaha commented 7 years ago

Hmm. Just to reconfirm, I am assuming that you have setup Voyager using https://github.com/appscode/voyager/blob/master/hack/deploy/with-rbac.yaml#L108 .

The logs above look ok to me. You can create the log level by setting the --v=5 . If that looks clean, you can increase it to --v=8. That will dump Kubernetes api calls. Though the logs will be too verbose at that point.

Also, which cloud provider are you using? Btw, we are available in our slack channel, https://slack.appscode.com/ to chat if you want to take a look.

tamalsaha commented 7 years ago

Hi, I see the problem. You did not set the value for the flags:

These flags are dependent on cloud provider. You can see examples values for various cloud providers here: https://github.com/appscode/voyager/tree/master/hack/deploy

tamalsaha commented 7 years ago

Please reopen, if you are still having this issue.

izutheo commented 7 years ago

Thanks that was it. We just forgot to delete this .... #need to salt this asap