Closed abbi-gaurav closed 5 years ago
Findings:
apiVersion: apps/v1
kind: Deployment
metadata:
name: in-memory-channel-dispatcher
namespace: knative-eventing
spec:
replicas: 1
selector:
matchLabels: &labels
clusterChannelProvisioner: in-memory-channel
role: dispatcher
template:
metadata:
annotations:
sidecar.istio.io/inject: "true"
labels: *labels
spec:
....
knative-serving test-service ExternalName <none> knative-ingressgateway.istio-system.svc.clu ster.local <none> 43m
knative-serving test-service-00001-service ClusterIP 10.96.32.240 <none> 80/TCP 44m
This generates updates error in the communication between Istio-pilot and Istio-sidecar for a pod running inside of Istio mesh:
[2019-01-21 08:45:16.943][20][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:223] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration update rejected: Onlyunique values for domains are permitted. Duplicate entry of domain test-service.knative-serving.svc.cluster.local
....
[2019-01-21 08:45:59.149][20][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:223] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration update rejected: Onlyunique values for domains are permitted. Duplicate entry of domain test-service.knative-serving.svc.cluster.local
Manual corrections were done to solve these two issues:
kubectl label namespace knative-eventing istio-injection=enabled
and deleting the "Knative-eventing dispatcher" pod recreate the KN dispatcher in Istio mesh.
After these changes, the original issue still remains and Istio mesh still remains in an unstable status, see the RDS column:
radufa@ubuntu:~/yaas/YSF/Go/src/github.com/kyma-project/kyma/components/event-bus/sample/knlib-docker$ sudo istioctl proxy-status
PROXY CDS LDS EDS RDS PILOT VERSION
activator-6cd459647f-4cbdj.knative-serving SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
activator-6cd459647f-dp68m.knative-serving SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
activator-6cd459647f-drbm9.knative-serving SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
application-registry-9dbbc7f88-bhh2t.kyma-integration SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
autoscaler-75b64694b-g7llm.knative-serving SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
configurations-generator-5fcd498684-cqxjl.kyma-system SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
connector-service-dcb886bcb-d7rvt.kyma-integration SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
core-binding-usage-controller-7b969b558c-tx64j.kyma-system SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
core-nats-streaming-0.kyma-system SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
core-publish-5987fbdb76-jxnkf.kyma-system SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
core-push-bd4fd657c-2tvwh.kyma-system SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
core-sub-validator-d8686f675-vmpc7.kyma-system SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
core-ui-api-567f958bc7-88q4r.kyma-system SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
in-memory-channel-dispatcher-54df589667-wn8d7.knative-eventing SYNCED SYNCED SYNCED (100%) STALE istio-pilot-55966fbdbc-gkjlq 1.0.2
istio-egressgateway-7cbdd7f856-w26l5.istio-system SYNCED SYNCED SYNCED (100%) NOT SENT istio-pilot-55966fbdbc-gkjlq 1.0.2
istio-ingressgateway-688865c5f7-kzbq8.istio-system SYNCED SYNCED SYNCED (100%) NOT SENT istio-pilot-55966fbdbc-gkjlq 1.0.2
knative-ingressgateway-59df55f8f6-mvklf.istio-system SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
knlib-657d6787dd-vdfx5.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-55966fbdbc-gkjlq 1.0.2
The virtualservice created by KN for the channel is:
apiVersion: v1
items:
- apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
clusterName: ""
creationTimestamp: 2019-01-21T08:40:33Z
generation: 1
labels:
channel: knlib-channel
provisioner: in-memory-channel
name: knlib-channel-channel
namespace: default
ownerReferences:
- apiVersion: eventing.knative.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Channel
name: knlib-channel
uid: 378a019f-1d58-11e9-9912-00155d966206
resourceVersion: "7603"
selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/knlib-channel-channel
uid: 379ca3f6-1d58-11e9-9912-00155d966206
spec:
hosts:
- knlib-channel-channel.default.svc.cluster.local
- knlib-channel.default.channels.cluster.local
http:
- rewrite:
authority: knlib-channel.default.channels.cluster.local
route:
- destination:
host: in-memory-channel-clusterbus.knative-eventing.svc.cluster.local
port:
number: 80
weight: 0
kind: List
metadata:
resourceVersion: ""
selfLink: ""
I've found the problem with injection in knative-eventing
. Knative have in their istio injection enabled by default in all namespaces. Accidentally we have similar change on our roadmap: https://app.zenhub.com/workspaces/kyma---all-repositories-5b6d5985084045741e744dea/issues/kyma-project/kyma/2073. This will solve the discrepancy.
The second problem is with serving tests. Somehow services created during tests are causing errors in pilot. Those services should be removed after tests. Issue: https://github.com/kyma-project/kyma/issues/2372
The second problem was solved by deleting the test directory from:
"./resources/knative/templates/test"
before starting the Kyma installation. After Kyma was installed, the communication between Pilot and Envoy was stable.
Summarizing, after the two found issues were manually fixed, the complete Kyma on Kantive is running, (inclusive Kyma Istio pacth).
Also our K8S test application which uses KN/Eventing without being a KN service is running and it can send messages to a KN channel.
Fix for tests just got merged (#2381). They are disabled now. There is a follow up to fix them instead of disabling (#2372), but it's not a blocker any longer.
Description While implementing a Publish served by Knative, it was discovered that a Publish service (a plain K8S service with Istio sidecar) cannot connect to Istio virtual service which represents a Knative eventing channel.
The flow is as below:
e.g.
error is :
Please refer https://github.com/kyma-project/community/blob/master/sig-and-wg/wg-knative/assets/event-types.svg and https://github.com/kyma-project/community/blob/master/sig-and-wg/wg-knative/assets/publish-api.svg for details.