search

openservicemesh / osm

Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.
https://openservicemesh.io/
Apache License 2.0
2.59k stars 277 forks source link

Working with an httpS server not supported ? #5324

Closed michaelcourcy closed 1 year ago

michaelcourcy commented 1 year ago

Bug description:

I am in permissive mode, here is my repro

I create an ngninx sever that listen on port 80 and 443 for https.

kubectl create ns nginx 
kubectl config set-context --current --namespace=nginx
helm upgrade nginx bitnami/nginx \
    --set service.type=ClusterIP \
    --set containerPorts.https=6443

I test by creating a curl pod 

kubectl run -it --rm --image=curlimages/curl curl --command -- /bin/sh
curl -k https://nginx

And I get the usual nginx default page 

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
....

If now I add nginx namespace to OSM and delete all the pods

osm namespace add nginx
kubectl delete po --all

I recreate a curl pod and try to access the https service 

kubectl run -it --rm --image=curlimages/curl curl --command -- /bin/sh

On port 80 it's fine 

curl  http://nginx
<!DOCTYPE html>
<html>
<head>
....

But on port 443 I get this error 

curl -kv https://nginx
*   Trying 10.0.6.59:443...
* Connected to nginx (10.0.6.59) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL/1.1.1t: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) OpenSSL/1.1.1t: error:1408F10B:SSL routines:ssl3_get_record:wrong version number

If I add the appProtocol field to the https port on the nginx service 

kubectl get svc nginx -o yaml 
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: nginx
    meta.helm.sh/release-namespace: nginx
  creationTimestamp: "2023-04-14T15:23:48Z"
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-13.2.33
  name: nginx
  namespace: nginx
  resourceVersion: "4230856"
  uid: f36731f4-4ea1-4477-9e4e-de29bdccacce
spec:
  clusterIP: 10.0.6.59
  clusterIPs:
  - 10.0.6.59
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: nginx
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Then I get another error

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to nginx:443

Did I miss a configuration here ?

Here is my ngninx pod 

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      cni.projectcalico.org/containerID: b79bc8d52d17110f9f6848e9443e6ce990446c82ccc63cc9d19261dc6c0c13ae
      cni.projectcalico.org/podIP: 10.244.4.30/32
      cni.projectcalico.org/podIPs: 10.244.4.30/32
    creationTimestamp: "2023-04-14T15:35:58Z"
    generateName: nginx-59dc55447-
    labels:
      app.kubernetes.io/instance: nginx
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: nginx
      helm.sh/chart: nginx-13.2.33
      osm-proxy-uuid: 61689ccb-c287-4544-8715-f52b411cf5cd
      pod-template-hash: 59dc55447
    name: nginx-59dc55447-wcfcs
    namespace: nginx
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: nginx-59dc55447
      uid: 47533d65-0e30-4386-b2d0-48c51feb446f
    resourceVersion: "4226593"
    uid: ef9863c5-b32f-412e-aeed-6f6d51fa1193
  spec:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/instance: nginx
                app.kubernetes.io/name: nginx
            topologyKey: kubernetes.io/hostname
          weight: 1
    automountServiceAccountToken: false
    containers:
    - env:
      - name: BITNAMI_DEBUG
        value: "false"
      - name: NGINX_HTTP_PORT_NUMBER
        value: "8080"
      - name: NGINX_HTTPS_PORT_NUMBER
        value: "6443"
      image: docker.io/bitnami/nginx:1.23.4-debian-11-r1
      imagePullPolicy: IfNotPresent
      livenessProbe:
        failureThreshold: 6
        httpGet:
          httpHeaders:
          - name: Original-Tcp-Port
            value: "8080"
          path: /osm-healthcheck
          port: 15904
          scheme: HTTP
        initialDelaySeconds: 30
        periodSeconds: 10
        successThreshold: 1
        timeoutSeconds: 5
      name: nginx
      ports:
      - containerPort: 8080
        name: http
        protocol: TCP
      - containerPort: 6443
        name: https
        protocol: TCP
      readinessProbe:
        failureThreshold: 3
        httpGet:
          httpHeaders:
          - name: Original-Tcp-Port
            value: "8080"
          path: /osm-healthcheck
          port: 15904
          scheme: HTTP
        initialDelaySeconds: 5
        periodSeconds: 5
        successThreshold: 1
        timeoutSeconds: 3
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    - args:
      - --verbosity
      - trace
      command:
      - /osm-healthcheck
      image: openservicemesh/osm-healthcheck@sha256:e6b3562d18bce78aa11acc641d7d55ac6b51e0b83fbd86eaaef2a56967f05f6b
      imagePullPolicy: IfNotPresent
      name: osm-healthcheck
      ports:
      - containerPort: 15904
        protocol: TCP
      resources: {}
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    - args:
      - --log-level
      - error
      - --config-path
      - /etc/envoy/bootstrap.yaml
      - --service-cluster
      - default.nginx
      command:
      - envoy
      env:
      - name: POD_UID
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.uid
      - name: POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      - name: POD_IP
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: status.podIP
      - name: SERVICE_ACCOUNT
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: spec.serviceAccountName
      image: envoyproxy/envoy-distroless:v1.22.2@sha256:541d31419b95e3c62d8cc0967db9cdb4ad2782cc08faa6f15f04c081200e324a
      imagePullPolicy: IfNotPresent
      name: envoy
      ports:
      - containerPort: 15000
        name: proxy-admin
        protocol: TCP
      - containerPort: 15003
        name: proxy-inbound
        protocol: TCP
      - containerPort: 15010
        name: proxy-metrics
        protocol: TCP
      - containerPort: 15901
        name: liveness-port
        protocol: TCP
      - containerPort: 15902
        name: readiness-port
        protocol: TCP
      resources: {}
      securityContext:
        allowPrivilegeEscalation: false
        runAsUser: 1500
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
      - mountPath: /etc/envoy
        name: envoy-bootstrap-config-volume
        readOnly: true
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    initContainers:
    - args:
      - -c
      - |
        iptables-restore --noflush <<EOF
        # OSM sidecar interception rules
        *nat
        :OSM_PROXY_INBOUND - [0:0]
        :OSM_PROXY_IN_REDIRECT - [0:0]
        :OSM_PROXY_OUTBOUND - [0:0]
        :OSM_PROXY_OUT_REDIRECT - [0:0]
        -A OSM_PROXY_IN_REDIRECT -p tcp -j REDIRECT --to-port 15003
        -A PREROUTING -p tcp -j OSM_PROXY_INBOUND
        -A OSM_PROXY_INBOUND -p tcp --dport 15010 -j RETURN
        -A OSM_PROXY_INBOUND -p tcp --dport 15901 -j RETURN
        -A OSM_PROXY_INBOUND -p tcp --dport 15902 -j RETURN
        -A OSM_PROXY_INBOUND -p tcp --dport 15903 -j RETURN
        -A OSM_PROXY_INBOUND -p tcp --dport 15904 -j RETURN
        -A OSM_PROXY_INBOUND -p tcp -j OSM_PROXY_IN_REDIRECT
        -A OSM_PROXY_OUT_REDIRECT -p tcp -j REDIRECT --to-port 15001
        -A OSM_PROXY_OUT_REDIRECT -p tcp --dport 15000 -j ACCEPT
        -A OUTPUT -p tcp -j OSM_PROXY_OUTBOUND
        -A OSM_PROXY_OUTBOUND -o lo ! -d 127.0.0.1/32 -m owner --uid-owner 1500 -j OSM_PROXY_IN_REDIRECT
        -A OSM_PROXY_OUTBOUND -o lo -m owner ! --uid-owner 1500 -j RETURN
        -A OSM_PROXY_OUTBOUND -m owner --uid-owner 1500 -j RETURN
        -A OSM_PROXY_OUTBOUND -d 127.0.0.1/32 -j RETURN
        -A OSM_PROXY_OUTBOUND -j OSM_PROXY_OUT_REDIRECT
        COMMIT
        EOF
      command:
      - /bin/sh
      env:
      - name: POD_IP
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: status.podIP
      image: openservicemesh/init@sha256:f0717caf400d81fb00e63e64710fb78d76da1d396155408b3fc56eeb80d6f0bd
      imagePullPolicy: IfNotPresent
      name: osm-init
      resources: {}
      securityContext:
        capabilities:
          add:
          - NET_ADMIN
        privileged: false
        runAsNonRoot: false
        runAsUser: 0
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    nodeName: aks-nodepool1-39371769-vmss000003
    preemptionPolicy: PreemptLowerPriority
    priority: 0
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: default
    serviceAccountName: default
    shareProcessNamespace: false
    terminationGracePeriodSeconds: 30
    tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    volumes:
    - name: envoy-bootstrap-config-volume
      secret:
        defaultMode: 420
        secretName: envoy-bootstrap-config-61689ccb-c287-4544-8715-f52b411cf5cd
  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: "2023-04-14T15:36:10Z"
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: "2023-04-14T15:36:18Z"
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: "2023-04-14T15:36:18Z"
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: "2023-04-14T15:35:58Z"
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: containerd://525cc2357aabd77bc0af34ff6f3d489fdeb6526ac97c63ebd7f6683c768fa8bc
      image: sha256:5992440fc58784182d65340f36733dac382259e00765468a7e09fd3000b2fec5
      imageID: docker.io/envoyproxy/envoy-distroless@sha256:541d31419b95e3c62d8cc0967db9cdb4ad2782cc08faa6f15f04c081200e324a
      lastState: {}
      name: envoy
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2023-04-14T15:36:10Z"
    - containerID: containerd://84f53d2524c59a154d77a6fd4b215b5f96c70d12d5516a7a56e9e934c7cfc055
      image: docker.io/bitnami/nginx:1.23.4-debian-11-r1
      imageID: docker.io/bitnami/nginx@sha256:93cbb1763e7a763756f6ffefda3ea38a2d1db5b8d383d7d67aa6b4a2933272ec
      lastState: {}
      name: nginx
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2023-04-14T15:36:10Z"
    - containerID: containerd://a3c5f028c24b3aa695a2754465b9b5d0dbb879bcf0b2469155afcf0e1c09d6f2
      image: sha256:fbbd0b36d3f6602c023c013f974bb7c70e3455e1058bc39f4dd4d6bb3c41b71c
      imageID: docker.io/openservicemesh/osm-healthcheck@sha256:e6b3562d18bce78aa11acc641d7d55ac6b51e0b83fbd86eaaef2a56967f05f6b
      lastState: {}
      name: osm-healthcheck
      ready: true
      restartCount: 0
      started: true
      state:
        running:
          startedAt: "2023-04-14T15:36:10Z"
    hostIP: 10.224.0.7
    initContainerStatuses:
    - containerID: containerd://a873530a0ae7762c99c8bae392cd060caebbda540edf496e4f2b7c731827ca39
      image: sha256:3e1dab5f8ddda38be9e5947d110269032ed676037bff1476e42114037db23b19
      imageID: docker.io/openservicemesh/init@sha256:f0717caf400d81fb00e63e64710fb78d76da1d396155408b3fc56eeb80d6f0bd
      lastState: {}
      name: osm-init
      ready: true
      restartCount: 0
      state:
        terminated:
          containerID: containerd://a873530a0ae7762c99c8bae392cd060caebbda540edf496e4f2b7c731827ca39
          exitCode: 0
          finishedAt: "2023-04-14T15:36:09Z"
          reason: Completed
          startedAt: "2023-04-14T15:36:09Z"
    phase: Running
    podIP: 10.244.4.30
    podIPs:
    - ip: 10.244.4.30
    qosClass: BestEffort
    startTime: "2023-04-14T15:35:58Z"
kind: List
metadata:
  resourceVersion: ""

Affected area (please mark with X where applicable):

Expected behavior:

Steps to reproduce the bug (as precisely as possible):

How was OSM installed?:

Anything else we need to know?:

Bug report archive:

Environment:

MESH NAME MESH NAMESPACE VERSION GIT COMMIT BUILD DATE osm osm-system v1.2.3 6ecc61cf9fb95270483ea8cc72af4a72e16a65ef 2022-12-14-01:23

- Kubernetes version (use `kubectl version`):

k version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.1", GitCommit:"8f94681cd294aa8cfd3407b8191f6c70214973a4", GitTreeState:"clean", BuildDate:"2023-01-18T15:51:24Z", GoVersion:"go1.19.5", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9", GitCommit:"57fbbcc2804848b95cad5519f5ec9d6355430db9", GitTreeState:"clean", BuildDate:"2023-02-08T17:22:38Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.26) and server (1.24) exceeds the supported minor version skew of +/-1


- Size of cluster (number of worker nodes in the cluster): 4
- Others:
michaelcourcy commented 1 year ago

I realize that my ask may not have sense, service mesh is here to inspect and control network traffic, if using https no inspection and decision can be done.

All that said is that possible to exclude a specific service in a namespace from the service mesh or does it have to be in another namespace ?

steeling commented 1 year ago

You can disable injection for specific pods

Can you try setting the appProtocl to "tcp" lowercase?

Can you run the osm osm verify connectivity command between the 2 pods?

michaelcourcy commented 1 year ago

Thanks @steeling I was able to disable injections for this specific pods.