Unable to configure sidecar TLS termination

juicemia commented 11 months ago

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description

I'm trying to forward traffic to my pods from a GCP load balancer. HTTP requests work fine, but gRPC requests are failing.

GCP requires that gRPC backends accept TLS connections, so I'm trying to set up TLS termination on the sidecars using this guide.

When I make a gRPC request, I get the following:

grpcurl -v -H "Authorization: Bearer $TOKEN" -import-path . -import-path api-common-protos -proto ./internal/inventory_item.proto $HOST internal.v1.InventoryItemService/ListInventoryItems

Resolved method descriptor:
// Gets all global or workspace inventory items,
rpc ListInventoryItems ( .internal.v1.ListInventoryItemsRequest ) returns ( .internal.v1.ListInventoryItemsResponse );

Request metadata to send:
authorization: Bearer REDACTED

Response headers received:
(empty)

Response trailers received:
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
content-length: 0
content-type: application/grpc
date: Mon, 27 Nov 2023 16:32:59 GMT
via: 1.1 google
Sent 0 requests and received 0 responses
ERROR:
  Code: Unavailable
  Message: upstream connect error or disconnect/reset before headers. reset reason: remote connection failure

I see the following log in the istio-proxy container:

{"hostname":"dotnet-api-5d484bf4d4-t4fph","response_flags":"-","protocol":null,"message":"dotnet-api-5d484bf4d4-t4fph -\u003e - 0 - - --","user_agent":null,"requested_server_name":null,"authority":null,"upstream_local_address":"127.0.0.6:53849","path":null,"upstream_host":"10.81.7.205:8081","response_code":0,"start_time":"2023-11-27T16:32:59.417Z","grpc_status":null,"method":null,"severity":"INFO"}

The sidecars I'm working with are on the latest version of Istio:

istioctl ps | grep -i dotnet
dotnet-api-5d484bf4d4-t4fph                      Kubernetes     SYNCED     SYNCED                         SYNCED     SYNCED       NOT SENT     istiod-7fdcb4587-g72mb     1.20.0
dotnet-api-5d484bf4d4-w9gzl                  Kubernetes     SYNCED     SYNCED                         SYNCED     SYNCED       NOT SENT     istiod-7fdcb4587-g72mb     1.20.0

At this point I don't know what else I can configure, I don't see any logs telling me anything is particularly wrong.

Version

$ istioctl version
client version: 1.20.0
control plane version: 1.20.0
data plane version: 1.14.6 (1 proxies), 1.16.7 (2 proxies), 1.20.0 (65 proxies)

$ k version
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.25.10-gke.2700

Additional Information

No response

juicemia commented 11 months ago

Also adding the config I ended up with:

Sidecar:

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: dotnet-api
spec:
  egress:
  - hosts:
    - REDACTED
  ingress:
  - defaultEndpoint: 0.0.0.0:8081
    port:
      name: grpc
      number: 8081
      protocol: TLS
    tls:
      caCertificates: /etc/istio/tls-ca-certs/ca.crt
      mode: SIMPLE
      privateKey: /etc/istio/tls-certs/tls.key
      serverCertificate: /etc/istio/tls-certs/tls.crt
  - defaultEndpoint: 0.0.0.0:80
    port:
      name: http
      number: 80
      protocol: HTTP
  outboundTrafficPolicy:
    mode: REGISTRY_ONLY
  workloadSelector:
    labels:
      app: dotnet-api

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: dotnet-api
    version: 0.1.1137
  name: dotnet-api
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: dotnet-api
  strategy:
    rollingUpdate:
      maxSurge: 50%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        sidecar.istio.io/userVolume: '{"tls-secret":{"secret":{"secretName":"ht-test-dotnet-api-grpc"}}}'
        sidecar.istio.io/userVolumeMount: '{"tls-secret":{"mountPath":"/etc/istio/tls-certs/","readOnly":true}}'
      creationTimestamp: null
      labels:
        app: dotnet-api
        version: 0.1.1137
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: dotnet-api
              topologyKey: failure-domain.beta.kubernetes.io/zone
            weight: 10
      automountServiceAccountToken: true
      containers:
        image: dotnet-api
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /__health
            port: 80
            scheme: HTTP
          initialDelaySeconds: 30
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 5
        name: dotnet-api
        ports:
        - containerPort: 8081
          name: grpc
          protocol: TCP
        - containerPort: 80
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /__health
            port: 80
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "3"
            memory: 1500M
          requests:
            cpu: 50m
            memory: 750M
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /elastic-certs
          mountPropagation: None
          name: elastic-certs
          readOnly: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      imagePullSecrets:
      - name: gcr-credentials
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: dotnet-api
      serviceAccountName: dotnet-api
      shareProcessNamespace: false
      terminationGracePeriodSeconds: 30
      volumes:
      - name: elastic-certs
        secret:
          defaultMode: 420
          optional: false
          secretName: app-ssl-es-http-certs-public

Service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"exposed_ports":{"80":{},"8081":{}}}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"80":"k8s1-aa1c883d-namespace-dotnet-api-80-2855fd52","8081":"k8s1-aa1c883d-namespace-dotnet-api-8081-2082cadf"},"zones":["europe-west3-a","europe-west3-b","europe-west3-c"]}'
  labels:
    app: dotnet-api
    version: 0.1.1137
  name: dotnet-api
spec:
  clusterIP: 10.82.96.194
  clusterIPs:
  - 10.82.96.194
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: HTTP2
    name: grpc-dotnet-api
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: http-dotnet-api
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: dotnet-api
  sessionAffinity: None
  type: ClusterIP

istio-policy-bot commented 8 months ago

🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2023-11-27. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.

Created by the issue and PR lifecycle manager.

istio / istio