search

kyma-project / kyma

Kyma is an opinionated set of Kubernetes-based modular building blocks, including all necessary capabilities to develop and run enterprise-grade cloud-native applications.
https://kyma-project.io
Apache License 2.0
1.51k stars 404 forks source link

Communication between a K8S Service and Istio virtual service is broken #2363

Closed abbi-gaurav closed 5 years ago

abbi-gaurav commented 5 years ago

Description While implementing a Publish served by Knative, it was discovered that a Publish service (a plain K8S service with Istio sidecar) cannot connect to Istio virtual service which represents a Knative eventing channel.

The flow is as below:

publish app (k8s deployment + istio side car ) ----> k8s service (channel) ---> istio virtual service (channel) ---> dispatcher service ---> dispatcher pod

e.g.

publish app --> foo-channel.svc.cluster.local --> foo-channel (Istio Virtual service) --> nats-dispatcher.knative-eventing.svc.cluster.local

error is : 

Route not found nats-dispatcher.knative-eventing.svc.cluster.local

Note: The same flow works on vanila K8s + Knative. This points to a possible discrepancy with Istio installation in Kyma

Please refer https://github.com/kyma-project/community/blob/master/sig-and-wg/wg-knative/assets/event-types.svg and https://github.com/kyma-project/community/blob/master/sig-and-wg/wg-knative/assets/publish-api.svg for details.

ghost commented 5 years ago

Findings:

  1. After installation of Kyma/Knative the "dispatcher" Knative eventing service was running without Istio sidecar. See in: https://github.com/knative/eventing/blob/master/config/provisioners/in-memory-channel/in-memory-channel.yaml 
    apiVersion: apps/v1
kind: Deployment
metadata:
name: in-memory-channel-dispatcher
namespace: knative-eventing
spec:
replicas: 1
selector:
matchLabels: &labels
  clusterChannelProvisioner: in-memory-channel
  role: dispatcher
template:
metadata:
  annotations:
    sidecar.istio.io/inject: "true"
  labels: *labels
spec:
....
  2. After the installation of Kyma/Knative was finished two "test" services remain running: 
    knative-serving    test-service                                          ExternalName   <none>           knative-ingressgateway.istio-system.svc.clu                   ster.local   <none>                                                                                                                    43m
knative-serving    test-service-00001-service                            ClusterIP      10.96.32.240     <none>                                                                     80/TCP                                                                                                                    44m

    This generates updates error in the communication between Istio-pilot and Istio-sidecar for a pod running inside of Istio mesh:

    [2019-01-21 08:45:16.943][20][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:223] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration update rejected: Onlyunique values for domains are permitted. Duplicate entry of domain test-service.knative-serving.svc.cluster.local
....
[2019-01-21 08:45:59.149][20][warning][upstream] external/envoy/source/common/config/grpc_mux_impl.cc:223] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration update rejected: Onlyunique values for domains are permitted. Duplicate entry of domain test-service.knative-serving.svc.cluster.local

    Manual corrections were done to solve these two issues:

    • for the first one, setting: 
      kubectl label namespace knative-eventing istio-injection=enabled

      and deleting the "Knative-eventing dispatcher" pod recreate the KN dispatcher in Istio mesh.

    • for the second one, manually deleting the two services solved the issue, so that envoy doesn't reject the "configuration update" requst.

After these changes, the original issue still remains and Istio mesh still remains in an unstable status, see the RDS column:

radufa@ubuntu:~/yaas/YSF/Go/src/github.com/kyma-project/kyma/components/event-bus/sample/knlib-docker$ sudo istioctl proxy-status
PROXY                                                              CDS        LDS        EDS               RDS          PILOT                            VERSION
activator-6cd459647f-4cbdj.knative-serving                         SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
activator-6cd459647f-dp68m.knative-serving                         SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
activator-6cd459647f-drbm9.knative-serving                         SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
application-registry-9dbbc7f88-bhh2t.kyma-integration              SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
autoscaler-75b64694b-g7llm.knative-serving                         SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
configurations-generator-5fcd498684-cqxjl.kyma-system              SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
connector-service-dcb886bcb-d7rvt.kyma-integration                 SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
core-binding-usage-controller-7b969b558c-tx64j.kyma-system         SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
core-nats-streaming-0.kyma-system                                  SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
core-publish-5987fbdb76-jxnkf.kyma-system                          SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
core-push-bd4fd657c-2tvwh.kyma-system                              SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
core-sub-validator-d8686f675-vmpc7.kyma-system                     SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
core-ui-api-567f958bc7-88q4r.kyma-system                           SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
in-memory-channel-dispatcher-54df589667-wn8d7.knative-eventing     SYNCED     SYNCED     SYNCED (100%)     STALE        istio-pilot-55966fbdbc-gkjlq     1.0.2
istio-egressgateway-7cbdd7f856-w26l5.istio-system                  SYNCED     SYNCED     SYNCED (100%)     NOT SENT     istio-pilot-55966fbdbc-gkjlq     1.0.2
istio-ingressgateway-688865c5f7-kzbq8.istio-system                 SYNCED     SYNCED     SYNCED (100%)     NOT SENT     istio-pilot-55966fbdbc-gkjlq     1.0.2
knative-ingressgateway-59df55f8f6-mvklf.istio-system               SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
knlib-657d6787dd-vdfx5.default                                     SYNCED     SYNCED     SYNCED (100%)     SYNCED       istio-pilot-55966fbdbc-gkjlq     1.0.2
ghost commented 5 years ago

The virtualservice created by KN for the channel is:

apiVersion: v1
items:
- apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
    clusterName: ""
    creationTimestamp: 2019-01-21T08:40:33Z
    generation: 1
    labels:
      channel: knlib-channel
      provisioner: in-memory-channel
    name: knlib-channel-channel
    namespace: default
    ownerReferences:
    - apiVersion: eventing.knative.dev/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: Channel
      name: knlib-channel
      uid: 378a019f-1d58-11e9-9912-00155d966206
    resourceVersion: "7603"
    selfLink: /apis/networking.istio.io/v1alpha3/namespaces/default/virtualservices/knlib-channel-channel
    uid: 379ca3f6-1d58-11e9-9912-00155d966206
  spec:
    hosts:
    - knlib-channel-channel.default.svc.cluster.local
    - knlib-channel.default.channels.cluster.local
    http:
    - rewrite:
        authority: knlib-channel.default.channels.cluster.local
      route:
      - destination:
          host: in-memory-channel-clusterbus.knative-eventing.svc.cluster.local
          port:
            number: 80
        weight: 0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
sjanota commented 5 years ago

I've found the problem with injection in knative-eventing. Knative have in their istio injection enabled by default in all namespaces. Accidentally we have similar change on our roadmap: https://app.zenhub.com/workspaces/kyma---all-repositories-5b6d5985084045741e744dea/issues/kyma-project/kyma/2073. This will solve the discrepancy.

sjanota commented 5 years ago

The second problem is with serving tests. Somehow services created during tests are causing errors in pilot. Those services should be removed after tests. Issue: https://github.com/kyma-project/kyma/issues/2372

ghost commented 5 years ago

The second problem was solved by deleting the test directory from:

"./resources/knative/templates/test"

before starting the Kyma installation. After Kyma was installed, the communication between Pilot and Envoy was stable.

ghost commented 5 years ago

Summarizing, after the two found issues were manually fixed, the complete Kyma on Kantive is running, (inclusive Kyma Istio pacth).

Also our K8S test application which uses KN/Eventing without being a KN service is running and it can send messages to a KN channel.

sjanota commented 5 years ago

Fix for tests just got merged (#2381). They are disabled now. There is a follow up to fix them instead of disabling (#2372), but it's not a blocker any longer.