search

knative / serving

Kubernetes-based, scale-to-zero, request-driven compute
https://knative.dev/docs/serving/
Apache License 2.0
5.54k stars 1.15k forks source link

Helloworld example get 503 #4281

Closed yuxiaoba closed 5 years ago

yuxiaoba commented 5 years ago

What version of Knative?

Knative 0.6

Expected Behavior

curl -H "Host: ${HOST_URL}" http://${IP_ADDRESS}
Hello World: Go Sample v1!

Actual Behavior

curl -H "Host: ${HOST_URL}" http://${IP_ADDRESS} -v
* Rebuilt URL to: http://192.168.199.181:31380/
*   Trying 192.168.199.181...
* Connected to 192.168.199.181 (192.168.199.181) port 31380 (#0)
> GET / HTTP/1.1
> Host: helloworld-go.default.example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< date: Thu, 06 Jun 2019 13:41:02 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 192.168.199.181 left intact

Steps to Reproduce the Problem

Install Kubernetes 1.13

kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Install Istio 1.1.3 by istio-lean.yaml

$ kubectl get pod -n istio-system 
NAME                                    READY   STATUS    RESTARTS   AGE
istio-ingressgateway-56b84b6f99-54cbr   1/1     Running   0          70m
istio-pilot-54b59c86dd-t4lgm            1/1     Running   0          70m
zipkin-5bf9dcc949-g6jdx                 1/1     Running   0          62m

kubectl get svc -n istio-system 
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
istio-ingressgateway   NodePort    10.68.225.165   <none>        15020:30437/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:34185/TCP,15030:20038/TCP,15031:34888/TCP,15032:23065/TCP,15443:21182/TCP   4m48s
istio-pilot            ClusterIP   10.68.168.40    <none>        15010/TCP,15011/TCP,8080/TCP,15014/TCP

I have changed the type of istio-ingressway from LoadBalancer to NodePort because my cluster is not support LoadBalancer

Install Knative 0.6

$ kubectl get pod --all-namespaces | grep knative
knative-build        build-controller-65455c5dd7-r5dv6               1/1     Running   0          70m
knative-build        build-webhook-6dc97d957d-bcmh5                  1/1     Running   0          70m
knative-eventing     eventing-controller-66df9ff95b-67gk6            1/1     Running   0          70m
knative-eventing     in-memory-channel-controller-6bcf6d98c9-qdn9j   1/1     Running   0          70m
knative-eventing     in-memory-channel-dispatcher-6dc47888fb-8hqst   1/1     Running   0          70m
knative-eventing     sources-controller-66c8477b87-bbskx             1/1     Running   0          70m
knative-eventing     webhook-f46db9dbd-g87xk                         1/1     Running   0          70m
knative-monitoring   elasticsearch-logging-0                         1/1     Running   0          70m
knative-monitoring   elasticsearch-logging-1                         1/1     Running   0          70m
knative-monitoring   grafana-848574f8dc-gtkh6                        1/1     Running   0          69m
knative-monitoring   kibana-logging-7698db4f94-8vvj6                 1/1     Running   0          70m
knative-monitoring   kube-state-metrics-5fcd59d4dd-zdczb             4/4     Running   0          49m
knative-monitoring   node-exporter-287zv                             2/2     Running   0          69m
knative-monitoring   node-exporter-4ns5v                             2/2     Running   0          69m
knative-monitoring   node-exporter-58k84                             2/2     Running   0          70m
knative-monitoring   node-exporter-88nn2                             2/2     Running   0          69m
knative-monitoring   node-exporter-9d6s6                             2/2     Running   0          69m
knative-monitoring   node-exporter-kxvwv                             2/2     Running   0          70m
knative-monitoring   node-exporter-lgq45                             2/2     Running   0          69m
knative-monitoring   node-exporter-lt8kb                             2/2     Running   0          69m
knative-monitoring   node-exporter-nv8m8                             2/2     Running   0          70m
knative-monitoring   node-exporter-sxlks                             2/2     Running   0          69m
knative-monitoring   prometheus-system-0                             1/1     Running   0          69m
knative-monitoring   prometheus-system-1                             1/1     Running   0          69m
knative-serving      activator-588b7df89b-s9vxn                      1/1     Running   0          70m
knative-serving      autoscaler-794f79d958-jc57w                     1/1     Running   0          70m
knative-serving      controller-5df4cf877f-gfsc2                     1/1     Running   0          70m
knative-serving      networking-istio-74b8d4ff9d-z2f46               1/1     Running   0          70m
knative-serving      webhook-79fcb54b99-z9rmj                        1/1     Running   0          70m
knative-sources      controller-manager-0                            1/1     Running   0          70m

Apply helloworld example like document

$ kubectl get ksvc
NAME            URL                                        LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.example.com   helloworld-go-7fgkw   helloworld-go-7fgkw   True

$ kubectl get svc $INGRESSGATEWAY --namespace istio-system
NAME                   TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)                                                                                                                                      AGE
istio-ingressgateway   NodePort   10.68.125.46   <none>        15020:25399/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:29766/TCP,15030:25294/TCP,15031:36330/TCP,15032:38266/TCP,15443:26655/TCP   79m

$ export IP_ADDRESS=$(kubectl get node  --output 'jsonpath={.items[0].status.addresses[0].address}'):$(kubectl get svc $INGRESSGATEWAY --namespace istio-system   --output 'jsonpath={.spec.ports[?(@.port==80)].nodePort}')
$ echo $IP_ADDRESS
192.168.199.181:31380
$ kubectl get route helloworld-go  --output=custom-columns=NAME:.metadata.name,DOMAIN:.status.domain
NAME            DOMAIN
helloworld-go   helloworld-go.default.example.com

$ curl -H "Host: helloworld-go.default.example.com" http://${IP_ADDRESS} -v
curl -H "Host: ${HOST_URL}" http://${IP_ADDRESS} -v
* Rebuilt URL to: http://192.168.199.181:31380/
*   Trying 192.168.199.181...
* Connected to 192.168.199.181 (192.168.199.181) port 31380 (#0)
> GET / HTTP/1.1
> Host: helloworld-go.default.example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< date: Thu, 06 Jun 2019 13:41:02 GMT
< server: istio-envoy
< content-length: 0
< 
* Connection #0 to host 192.168.199.181 left intact
yuxiaoba commented 5 years ago

The logs of istio

$ kubectl logs -n istio-system  istio-ingressgateway-56b84b6f99-54cbr
......
2019-06-06T12:37:44.938550Z info    Envoy proxy is ready
[2019-06-06 13:05:32.183][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, 
[2019-06-06 13:34:33.102][13][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, 

$ kubectl logs -n istio-system istio-pilot-54b59c86dd-t4lgm
....
2019-06-06T13:34:33.545427Z info    1 error occurred:

* gateway omitting listener "0.0.0.0_443" due to: must have more than 0 chains in listener: &v2.Listener{Name:"0.0.0.0_443", Address:core.Address{Address:(*core.Address_SocketAddress)(0xc420d33bf8), XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}, FilterChains:[]listener.FilterChain{}, UseOriginalDst:nil, PerConnectionBufferLimitBytes:nil, Metadata:(*core.Metadata)(nil), DeprecatedV1:(*v2.Listener_DeprecatedV1)(nil), DrainType:0, ListenerFilters:[]listener.ListenerFilter(nil), ListenerFiltersTimeout:(*time.Duration)(nil), Transparent:nil, Freebind:nil, SocketOptions:[]*core.SocketOption(nil), TcpFastOpenQueueLength:nil, XXX_NoUnkeyedLiteral:struct {}{}, XXX_unrecognized:[]uint8(nil), XXX_sizecache:0}
2019-06-06T13:34:33.545543Z info    ads LDS: PUSH for node:istio-ingressgateway-56b84b6f99-54cbr.istio-system addr:"172.20.7.52:34790" listeners:1 912
2019-06-06T13:34:33.546242Z info    ads ADS: RDS: PUSH for node: istio-ingressgateway-56b84b6f99-54cbr.istio-system addr:172.20.7.52:34790 routes:1
scvg23: inuse: 14, idle: 2, sys: 17, released: 0, consumed: 17 (MB)
gc 108 @3612.067s 0%: 0.040+10+0.073 ms clock, 0.080+1.8/3.5/9.4+0.14 ms cpu, 13->13->6 MB, 14 MB goal, 2 P
gc 109 @3649.538s 0%: 0.084+12+0.054 ms clock, 0.16+0.53/2.2/17+0.10 ms cpu, 13->13->6 MB, 14 MB goal, 2 P
gc 110 @3687.071s 0%: 0.017+14+0.057 ms clock, 0.034+2.5/11/13+0.11 ms cpu, 13->13->6 MB, 14 MB goal, 2 P
gc 111 @3725.978s 0%: 0.014+7.0+0.065 ms clock, 0.029+1.0/0.84/11+0.13 ms cpu, 13->13->6 MB, 14 MB goal, 2 P
yuxiaoba commented 5 years ago

The logs of Knative serving controller show Reconcile failed.

$ kubectl logs -n knative-serving  controller-5df4cf877f-gfsc2
...
{"level":"info","ts":"2019-06-06T13:47:46.417Z","logger":"controller.route-controller","caller":"route/route.go:419","msg":"Revision default/helloworld-go-7fgkw is inactive","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-06T13:47:46.418Z","logger":"controller.route-controller","caller":"route/route.go:441","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-06T13:47:46.418Z","logger":"controller.route-controller","caller":"route/route.go:296","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-06T13:47:46.419Z","logger":"controller.route-controller","caller":"route/route.go:319","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-06T13:47:46.420Z","logger":"controller.route-controller","caller":"route/route.go:356","msg":"Creating ClusterIngress.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-06T13:47:46.547Z","logger":"controller.route-controller","caller":"route/route.go:367","msg":"Updating placeholder k8s services with clusterIngress information","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}
{"level":"error","ts":"2019-06-06T13:47:46.663Z","logger":"controller.route-controller","caller":"controller/controller.go:311","msg":"Reconcile error","knative.dev/controller":"route-controller","error":"Service \"helloworld-go\" is invalid: spec.externalName: Invalid value: \"istio-ingressgateway.istio-system.svc.cluster.local.\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","stacktrace":"github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:311\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:297\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248"}
{"level":"info","ts":"2019-06-06T13:47:46.664Z","logger":"controller.route-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"0a7a5caf-885a-11e9-a9d7-5254004f8e0e\", APIVersion:\"serving.knative.dev/v1alpha1\", ResourceVersion:\"6175829\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' Service \"helloworld-go\" is invalid: spec.externalName: Invalid value: \"istio-ingressgateway.istio-system.svc.cluster.local.\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')","knative.dev/controller":"route-controller"}
{"level":"info","ts":"2019-06-06T13:47:46.666Z","logger":"controller.route-controller","caller":"controller/controller.go:298","msg":"Reconcile failed. Time taken: 253.007054ms.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}

The logs of networking-istio

$ kubectl logs -n knative-serving  networking-istio-6d95d868fb-gqlmf
{"level":"info","ts":"2019-06-17T09:14:33.725Z","logger":"controller-clusteringress-istio.clusteringress-controller","caller":"clusteringress/clusteringress.go:222","msg":"Reconciling clusterIngress: &v1alpha1.ClusterIngress{TypeMeta:v1.TypeMeta{Kind:\"\", APIVersion:\"\"}, ObjectMeta:v1.ObjectMeta{Name:\"route-468f0f7d-90e0-11e9-b455-5254004f8e0e\", GenerateName:\"\", Namespace:\"\", SelfLink:\"/apis/networking.internal.knative.dev/v1alpha1/clusteringresses/route-468f0f7d-90e0-11e9-b455-5254004f8e0e\", UID:\"51c49b68-90e0-11e9-b455-5254004f8e0e\", ResourceVersion:\"1434124\", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63696359673, loc:(*time.Location)(0x25ee000)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"serving.knative.dev/route\":\"helloworld-go\", \"serving.knative.dev/routeNamespace\":\"default\"}, Annotations:map[string]string{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"serving.knative.dev/v1alpha1\\\",\\\"kind\\\":\\\"Service\\\",\\\"metadata\\\":{\\\"annotations\\\":{},\\\"name\\\":\\\"helloworld-go\\\",\\\"namespace\\\":\\\"default\\\"},\\\"spec\\\":{\\\"template\\\":{\\\"spec\\\":{\\\"containers\\\":[{\\\"env\\\":[{\\\"name\\\":\\\"TARGET\\\",\\\"value\\\":\\\"Go Sample v1\\\"}],\\\"image\\\":\\\"dockerhub.azk8s.cn/tcnghia/helloworld-go\\\"}]}}}}\\n\", \"networking.knative.dev/ingress.class\":\"istio.ingress.networking.knative.dev\", \"serving.knative.dev/creator\":\"admin\", \"serving.knative.dev/lastModifier\":\"admin\"}, OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:\"\"}, Spec:v1alpha1.IngressSpec{DeprecatedGeneration:0, TLS:[]v1alpha1.ClusterIngressTLS(nil), Rules:[]v1alpha1.ClusterIngressRule{v1alpha1.ClusterIngressRule{Hosts:[]string{\"helloworld-go.default.example.com\", \"helloworld-go.default.svc.cluster.local.\"}, HTTP:(*v1alpha1.HTTPClusterIngressRuleValue)(0xc00088f2c0)}}, Visibility:\"ExternalIP\"}, Status:v1alpha1.IngressStatus{Status:v1beta1.Status{ObservedGeneration:1, Conditions:v1beta1.Conditions{apis.Condition{Type:\"LoadBalancerReady\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63696359673, loc:(*time.Location)(0x25ee000)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"NetworkConfigured\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63696359673, loc:(*time.Location)(0x25ee000)}}}, Reason:\"\", Message:\"\"}, apis.Condition{Type:\"Ready\", Status:\"True\", Severity:\"\", LastTransitionTime:apis.VolatileTime{Inner:v1.Time{Time:time.Time{wall:0x0, ext:63696359673, loc:(*time.Location)(0x25ee000)}}}, Reason:\"\", Message:\"\"}}}, LoadBalancer:(*v1alpha1.LoadBalancerStatus)(0xc00088f2e0)}}","knative.dev/controller":"clusteringress-controller","knative.dev/key":"route-468f0f7d-90e0-11e9-b455-5254004f8e0e"}
{"level":"info","ts":"2019-06-17T09:14:33.725Z","logger":"controller-clusteringress-istio.clusteringress-controller","caller":"clusteringress/clusteringress.go:223","msg":"Creating/Updating VirtualService","knative.dev/controller":"clusteringress-controller","knative.dev/key":"route-468f0f7d-90e0-11e9-b455-5254004f8e0e"}
{"level":"info","ts":"2019-06-17T09:14:33.726Z","logger":"controller-clusteringress-istio.clusteringress-controller","caller":"clusteringress/clusteringress.go:274","msg":"ClusterIngress successfully synced","knative.dev/controller":"clusteringress-controller","knative.dev/key":"route-468f0f7d-90e0-11e9-b455-5254004f8e0e"}
{"level":"info","ts":"2019-06-17T09:14:33.726Z","logger":"controller-clusteringress-istio.clusteringress-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 1.033939ms.","knative.dev/controller":"clusteringress-controller","knative.dev/key":"route-468f0f7d-90e0-11e9-b455-5254004f8e0e"}

The logs of autoscale

{"level":"info","ts":"2019-06-17T09:14:32.237Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 425.892µs.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:14:32.363Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=1, want=1","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:14:32.363Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 493.353µs.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:14:33.033Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 59.662µs.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go"}
{"level":"info","ts":"2019-06-17T09:15:32.365Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=1, want=0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.365Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:350","msg":"SKS changed; reconciling:helloworld-go-z2xgt","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.637Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 273.943814ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.637Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:350","msg":"SKS changed; reconciling:helloworld-go-z2xgt","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"error","ts":"2019-06-17T09:15:32.784Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:311","msg":"Reconcile error","knative.dev/controller":"kpa-class-podautoscaler-controller","error":"error reconciling SKS: error updating SKS helloworld-go-z2xgt: Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"helloworld-go-z2xgt\": the object has been modified; please apply your changes to the latest version and try again","errorVerbose":"Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"helloworld-go-z2xgt\": the object has been modified; please apply your changes to the latest version and try again\nerror updating SKS helloworld-go-z2xgt\ngithub.com/knative/serving/pkg/reconciler/autoscaling/kpa.(*Reconciler).reconcileSKS\n\t/go/src/github.com/knative/serving/pkg/reconciler/autoscaling/kpa/kpa.go:352\ngithub.com/knative/serving/pkg/reconciler/autoscaling/kpa.(*Reconciler).reconcile\n\t/go/src/github.com/knative/serving/pkg/reconciler/autoscaling/kpa/kpa.go:245\ngithub.com/knative/serving/pkg/reconciler/autoscaling/kpa.(*Reconciler).Reconcile\n\t/go/src/github.com/knative/serving/pkg/reconciler/autoscaling/kpa/kpa.go:206\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:296\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337\nerror reconciling SKS\ngithub.com/knative/serving/pkg/reconciler/autoscaling/kpa.(*Reconciler).reconcile\n\t/go/src/github.com/knative/serving/pkg/reconciler/autoscaling/kpa/kpa.go:247\ngithub.com/knative/serving/pkg/reconciler/autoscaling/kpa.(*Reconciler).Reconcile\n\t/go/src/github.com/knative/serving/pkg/reconciler/autoscaling/kpa/kpa.go:206\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:296\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1337","stacktrace":"github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).handleErr\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:311\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:297\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248"}
{"level":"info","ts":"2019-06-17T09:15:32.785Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:298","msg":"Reconcile failed. Time taken: 148.170791ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.787Z","logger":"autoscaler.kpa-class-podautoscaler-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"PodAutoscaler\", Namespace:\"default\", Name:\"helloworld-go-z2xgt\", UID:\"4e599a3d-90e0-11e9-b455-5254004f8e0e\", APIVersion:\"autoscaling.internal.knative.dev/v1alpha1\", ResourceVersion:\"1434105\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' error reconciling SKS: error updating SKS helloworld-go-z2xgt: Operation cannot be fulfilled on serverlessservices.networking.internal.knative.dev \"helloworld-go-z2xgt\": the object has been modified; please apply your changes to the latest version and try again","knative.dev/controller":"kpa-class-podautoscaler-controller"}
{"level":"info","ts":"2019-06-17T09:15:32.791Z","logger":"autoscaler","caller":"kpa/scaler.go:196","msg":"helloworld-go-z2xgt probing activator = true, err = <nil>"}
{"level":"info","ts":"2019-06-17T09:15:32.791Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=0, want=0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.791Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 5.517388ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.795Z","logger":"autoscaler","caller":"kpa/scaler.go:196","msg":"helloworld-go-z2xgt probing activator = true, err = <nil>"}
{"level":"info","ts":"2019-06-17T09:15:32.795Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=0, want=0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:15:32.795Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 3.933478ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:16:02.803Z","logger":"autoscaler","caller":"kpa/scaler.go:196","msg":"helloworld-go-z2xgt probing activator = true, err = <nil>"}
{"level":"info","ts":"2019-06-17T09:16:02.804Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/scaler.go:284","msg":"Scaling from 1 to 0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:16:02.879Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=0, want=0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:16:02.880Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 81.791014ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:16:03.399Z","logger":"autoscaler","caller":"kpa/scaler.go:196","msg":"helloworld-go-z2xgt probing activator = true, err = <nil>"}
{"level":"info","ts":"2019-06-17T09:16:03.399Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"kpa/kpa.go:281","msg":"PA scale got=0, want=0","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
{"level":"info","ts":"2019-06-17T09:16:03.400Z","logger":"autoscaler.kpa-class-podautoscaler-controller","caller":"controller/controller.go:305","msg":"Reconcile succeeded. Time taken: 3.463174ms.","knative.dev/controller":"kpa-class-podautoscaler-controller","knative.dev/key":"default/helloworld-go-z2xgt"}
vagababov commented 5 years ago 

invalid: spec.externalName: Invalid value:
\"istio-ingressgateway.istio-system.svc.cluster.local.

I think you have a trailing period there, which should not be there.

On Thu, Jun 6, 2019 at 6:58 AM yuxiaoba notifications@github.com wrote:

The logs of Knative serving controller

$ kubectl logs -n knative-serving controller-5df4cf877f-gfsc2 ... {"level":"info","ts":"2019-06-06T13:47:46.417Z","logger":"controller.route-controller","caller":"route/route.go:419","msg":"Revision default/helloworld-go-7fgkw is inactive","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"info","ts":"2019-06-06T13:47:46.418Z","logger":"controller.route-controller","caller":"route/route.go:441","msg":"All referred targets are routable, marking AllTrafficAssigned with traffic information.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"info","ts":"2019-06-06T13:47:46.418Z","logger":"controller.route-controller","caller":"route/route.go:296","msg":"Updating targeted revisions.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"info","ts":"2019-06-06T13:47:46.419Z","logger":"controller.route-controller","caller":"route/route.go:319","msg":"Creating placeholder k8s services","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"info","ts":"2019-06-06T13:47:46.420Z","logger":"controller.route-controller","caller":"route/route.go:356","msg":"Creating ClusterIngress.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"info","ts":"2019-06-06T13:47:46.547Z","logger":"controller.route-controller","caller":"route/route.go:367","msg":"Updating placeholder k8s services with clusterIngress information","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"} {"level":"error","ts":"2019-06-06T13:47:46.663Z","logger":"controller.route-controller","caller":"controller/controller.go:311","msg":"Reconcile error","knative.dev/controller":"route-controller","error":"Service \"helloworld-go\" is invalid: spec.externalName: Invalid value: \"istio-ingressgateway.istio-system.svc.cluster.local.\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(\.a-z0-9?)')","stacktrace":"github.com/knative/serving/vendor/github.com/knative/pkg/controller.(Impl).handleErr\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:311\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(Impl).processNextWorkItem\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:297\ngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(Impl).Run.func1\n\t/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248 http://github.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).handleErr%5Cn%5Ct/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:311%5Cngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).processNextWorkItem%5Cn%5Ct/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:297%5Cngithub.com/knative/serving/vendor/github.com/knative/pkg/controller.(*Impl).Run.func1%5Cn%5Ct/go/src/github.com/knative/serving/vendor/github.com/knative/pkg/controller/controller.go:248"} {"level":"info","ts":"2019-06-06T13:47:46.664Z","logger":"controller.route-controller.event-broadcaster","caller":"record/event.go:221","msg":"Event(v1.ObjectReference{Kind:\"Route\", Namespace:\"default\", Name:\"helloworld-go\", UID:\"0a7a5caf-885a-11e9-a9d7-5254004f8e0e\", APIVersion:\"serving.knative.dev/v1alpha1\ http://serving.knative.dev/v1alpha1%5C", ResourceVersion:\"6175829\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' Service \"helloworld-go\" is invalid: spec.externalName: Invalid value: \"istio-ingressgateway.istio-system.svc.cluster.local.\": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(\.a-z0-9?)*')","knative.dev/controller":"route-controller"} {"level":"info","ts":"2019-06-06T13:47:46.666Z","logger":"controller.route-controller","caller":"controller/controller.go:298","msg":"Reconcile failed. Time taken: 253.007054ms.","knative.dev/controller":"route-controller","knative.dev/key":"default/helloworld-go"}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/knative/serving/issues/4281?email_source=notifications&email_token=AAF2WX57LAWARI4JZZ65ZBTPZEJ2DA5CNFSM4HVCFEUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXC5RHY#issuecomment-499505311, or mute the thread https://github.com/notifications/unsubscribe-auth/AAF2WX3RVWNCDYCAXBO66WDPZEJ2DANCNFSM4HVCFEUA .

yuxiaoba commented 5 years ago

@vagababov Thank you for your help，could you tell me how to remove the trailing period?

vagababov commented 5 years ago

Do you have some customization on your knative deployment? Or changes to your istio deployment?

houshengbo commented 5 years ago

Try running the command "kubectl get cm config-istio -n knative-serving -o yaml" to see the value for "gateway.knative-ingress-gateway". it should be set to "istio-ingressgateway.istio-system.svc.cluster.local".

If you install from source code: you can edit the value for "gateway.knative-ingress-gateway" in file config/config-istio.yaml. Then apply it by kubectl apply -f config/config-istio.yaml.

However, another thing to remind: has your kube service already offered an existing domain name fro all services? If so, you have to use that domain name instead of example.com.

yuxiaoba commented 5 years ago

@houshengbo 

    gateway.knative-ingress-gateway: "istio-ingressgateway.istio-system.svc.cluster.local"
    local-gateway.cluster-local-gateway: "cluster-local-gateway.istio-system.svc.cluster.local"
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"_example":"################################\n#                              #\n#    EXAMPLE CONFIGURATION     #\n#                              #\n################################\n\n# This block is not actually functional configuration,\n# but serves to illustrate the available configuration\n# options and document them in a way that is accessible\n# to users that `kubectl edit` this config map.\n#\n# These sample configuration options may be copied out of\n# this example block and unindented to be in the data block\n# to actually change the configuration.\n\n# Default Knative Gateway after v0.3. It points to the Istio\n# standard istio-ingressgateway, instead of a custom one that we\n# used pre-0.3.\ngateway.knative-ingress-gateway: \"istio-ingressgateway.istio-system.svc.cluster.local\"\n\n# A cluster local gateway to allow pods outside of the mesh to access\n# Services and Routes not exposing through an ingress.  If the users\n# do have a service mesh setup, this isn't required and can be removed.\n#\n# An example use case is when users want to use Istio without any\n# sidecar injection (like Knative's istio-lean.yaml).  Since every pod\n# is outside of the service mesh in that case, a cluster-local  service\n# will need to be exposed to a cluster-local gateway to be accessible.\nlocal-gateway.cluster-local-gateway: \"cluster-local-gateway.istio-system.svc.cluster.local\"\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"networking.knative.dev/ingress-provider":"istio","serving.knative.dev/release":"v0.6.0"},"name":"config-istio","namespace":"knative-serving"}}
  creationTimestamp: "2019-06-06T12:39:24Z"
  labels:
    networking.knative.dev/ingress-provider: istio
    serving.knative.dev/release: v0.6.0
  name: config-istio
  namespace: knative-serving
  resourceVersion: "6173076"
  selfLink: /api/v1/namespaces/knative-serving/configmaps/config-istio
  uid: 1d3ecd81-8858-11e9-a9d7-5254004f8e0e

It seems the value is normal.

I don's remember I have offered an existing domain name for services， how can I see this?

yuxiaoba commented 5 years ago

I have change the type of istio-ingressway from LoadBalancer to NodePort because my cluster is not support LoadBalancer @vagababov

yuxiaoba commented 5 years ago

@gyliu513

yuxiaoba commented 5 years ago

@mattmoor The #4407 is install istio with sidecar injector. I also try to install istio without sidecar injector. But I also have another error, can you help to check it?

tcnghia commented 5 years ago

@yuxiaoba I'd like to take a closer look at the state of your cluster to see what's wrong here. Can you please join our Slack channel (https://knative.slack.com/)

yuxiaoba commented 5 years ago

@tcnghia thank you for your help， I am sorry to reply late for there is a time difference between us. I will join the slack and contact with you.

tcnghia commented 5 years ago

@yuxiaoba I think the logic in here didn't handle /etc/resolv.conf correctly for your cluster

https://github.com/knative/serving/blob/master/pkg/network/domain.go#L60

mattmoor commented 5 years ago

@tcnghia should this be fixed at HEAD?