search

aws / aws-app-mesh-examples

AWS App Mesh is a service mesh that you can use with your microservices to manage service to service communication.
MIT No Attribution
864 stars 395 forks source link

[BUG] annotation 'appmesh.k8s.aws/mesh' dose not work #230

Closed XiaoYangZhu closed 5 years ago

XiaoYangZhu commented 5 years ago

Important note on security disclosures: If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly.

Describe the bug I created a service mesh stack based on samples in the eskworkshop [https://eksworkshop.com/servicemesh_with_appmesh/] but this service mesh stack is created in another mesh in appmesh, which is not the default one used in injector configuration process("dj-app" for "APPMESH_NAME" in file .../2_create_injector/inject.yaml). All the resource is created successfully but when I enter into the dj container and executed the command "curl http://jazz.prod.svc.cluster.local:9080/;echo;" , the error prompted as "curl: (7) Failed to connect to jazz.dj-mesh-ns.svc.cluster.local port 9080: Connection refused".

Platform EKS 1.14

To Reproduce Steps to reproduce the behavior:

  1. kubectl apply -f dj-app-all.yaml
  2. kubectl get all -n dj-mesh-ns

NAME READY STATUS RESTARTS AGE pod/dj-79fdb9d967-fzgtv 2/2 Running 0 16h pod/jazz-v1-5cdbbdd469-trcsk 2/2 Running 0 16h pod/metal-v1-ffdf7cc54-mkc5n 2/2 Running 0 16h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/dj ClusterIP 10.100.106.91 9080/TCP 16h service/jazz ClusterIP 10.100.4.141 9080/TCP 16h service/jazz-v1 ClusterIP 10.100.116.162 9080/TCP 16h service/metal ClusterIP 10.100.19.226 9080/TCP 16h service/metal-v1 ClusterIP 10.100.150.52 9080/TCP 16h

NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/dj 1/1 1 1 16h deployment.apps/jazz-v1 1/1 1 1 16h deployment.apps/metal-v1 1/1 1 1 16h

NAME DESIRED CURRENT READY AGE replicaset.apps/dj-79fdb9d967 1 1 1 16h replicaset.apps/jazz-v1-5cdbbdd469 1 1 1 16h replicaset.apps/metal-v1-ffdf7cc54 1 1 1 16h

NAME AGE virtualnode.appmesh.k8s.aws/dj 17h virtualnode.appmesh.k8s.aws/jazz 17h virtualnode.appmesh.k8s.aws/jazz-v1 17h virtualnode.appmesh.k8s.aws/metal 17h virtualnode.appmesh.k8s.aws/metal-v1 17h

NAME AGE virtualservice.appmesh.k8s.aws/jazz.dj-mesh-ns.svc.cluster.local 16h virtualservice.appmesh.k8s.aws/metal.dj-mesh-ns.svc.cluster.local 16h

  1. kubectl exec -n dj-mesh-ns -it dj-79fdb9d967-fzgtv -c dj bash root@dj-79fdb9d967-fzgtv:/usr/src/app# curl jazz.dj-mesh-ns.svc.cluster.local:9080 curl: (7) Failed to connect to jazz.dj-mesh-ns.svc.cluster.local port 9080: Connection refused root@dj-79fdb9d967-fzgtv:/usr/src/app#

Expected behavior I should be able to sucessfully get some result like: curl http://jazz.prod.svc.cluster.local:9080/;echo; ["Astrud Gilberto","Miles Davis"]

Config files, and API responses The content of "dj-app-all.yaml" file is: dj-app-all.yaml.zip

Additional context I follow all the prerequisite required in the eksworkshop for appmesh above and also could successfully run the end to end sample using the default mesh "dj-app" used in the injector creation process but failed with the same sample content in the new mesh scenario.

kiranmeduri commented 5 years ago

Thank you for reporting this issue. Before we dive into troubleshooting this note that directions in eksworkshop are outdated as installing controller and injector is now driven via helm see https://github.com/aws/eks-charts. Apologies for the confusion, we will update the workshop asap to avoid this problem. See if you can try https://github.com/aws/aws-app-mesh-examples/tree/master/walkthroughs/eks as that is current.

To continue troubleshooting this bug please provide following information.

  1. You mentioned All the resource is created successfully. Does that mean you can use awscli to describe and find that resources are created properly in App Mesh?

  2. If so, can you provide output of following 

    kubectl exec -n dj-mesh-ns -it dj-79fdb9d967 -c envoy curl http://localhost:9901/clusters
(and)
kubectl exec -n dj-mesh-ns -it dj-79fdb9d967 -c envoy curl http://localhost:9901/listeners

    These commands will let us know if Envoy is getting any configuration from App Mesh.

  3. If the result of above commands is missing details about jazz and metal, then most likely Envoy is failing to connect. To find the reason check the logs of envoy container.

    kubectl logs -n dj-mesh-ns dj-79fdb9d967 -c envoy
kiranmeduri commented 5 years ago

I can confirm that this is due to old aws-app-mesh-injector installation. Currently eks-workshop is referring to aws-app-mesh-injector:0.1.0. Support for multiple meshes is added in 0.1.1.

For now you can upgrade your cluster to use latest aws-app-mesh-controller-for-k8s and aws-app-mesh-injector using the steps described in https://github.com/aws/eks-charts.

XiaoYangZhu commented 5 years ago

thank you and I will take a new try based on the latest info you provided here.