Helm Chart is generating incorrect Istio Objects - Policy is deprecated from Istio v1.4/5

stevejr commented 4 years ago

When using the OpenFaas helm chart with the istio.mtls=true value set the generated Istio objects are not compatible with newer Istio releases. The Policy object the helm chart is currently generating has been deprecated in favour of the PeerAuthentication object.

Expected Behaviour

The OpenFaaS helm chart should generate the correct Istio objects.

Current Behaviour

The OpenFaaS helm chart generats the incorrect Istio Policy objects.

Possible Solution

Amend the chart to default to creating PeerAuthentication objects i.e.

kind: "PeerAuthentication"
metadata:
name: "default"
namespace: "openfaas"
spec:
mtls:
mode: STRICT
---
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
name: "default"
namespace: "openfaas-fn"
spec:
mtls:
mode: STRICT

Amend the chart to take an additional Istio version value and then determine if it should create Istio Policy or PeerAuthentication objects

Steps to Reproduce (for bugs)

helm upgrade openfaas --install openfaas/openfaas \ --namespace openfaas \ --set functionNamespace=openfaas-fn \ --set generateBasicAuth=true \ --set istio.mtls=true

Context

Cannot use Istio mTLS with the OpenFaaS deployment.

Your Environment

FaaS-CLI version ( Full output from: faas-cli version ): CLI: commit: c12d57c39ac4cc6eef3c9bba2fb45113d882432f version: 0.12.14

Gateway uri: https://...... version: 0.20.1 sha: 7b6cc60bd9865852cd11c98d4420752815052918 commit: Fix Invocation count wrong for multiple namespace

Provider name: faas-netes orchestration: kubernetes version: 0.12.8 sha: e88c54c3854614fe02809c6931d544f74d64c6f0

Docker version docker version (e.g. Docker 17.0.05 ):
What version and distriubtion of Kubernetes are you using? kubectl version Kubernetes v1.18.8
Operating System and version (e.g. Linux, Windows, MacOS): Linux
Link to your project or a code example to reproduce issue:
What network driver are you using and what CIDR? i.e. Weave net / Flannel Calico

alexellis commented 4 years ago

Can you fill out a number of suggestions under Possible Solution please?

howardjohn commented 3 years ago

@alexellis do you have a requirement to support Istio 1.4 users? Note that Istio has been out of support for almost a year: https://istio.io/latest/news/support/announcing-1.4-eol-final/.

If not, it should be pretty simple to update, i can send a PR.

alexellis commented 3 years ago

Hi @howardjohn,

Thank you for reaching out :+1: - your help would be really appreciated.

Whilst we have a broad understanding of service-mesh and hands-on experience with Linkerd, the person who added and maintained the configurations for Istio cannot help with this any more and the alternative is removing Istio instructions.

I've reached out to Craig Box a few times to see if folks on the Istio team could help us bring the configs back up to date. OpenFaaS isn't using any particularly advanced or esoteric features of Istio and doesn't integrate to the API directly, but users have asked for a working configuration for mTLS.

OpenFaaS can be installed with arkade or helm (arkade is easier) https://docs.openfaas.com/deployment/kubernetes/ - just like Knative, each workload container i.e. "function" is deployed as a Deployment, and then a Pod.

The direct_functions / directFunctions flag has to be set to set to true to use a service mesh, so that the gateway uses the Service name of the function, if not, the endpoints are looked up and randomly accessed by endpoint IP (which I think would invalidate the use of mTLS, or just not work)

We also have settings to switch to and from HTTP and exec probes. Does Istio still require exec probes? (HTTP tend to be more efficient)

A PR would be great, if you are able to give some broad instructions for deploying it with Istio, that would help us verify the behaviour too.

If you're open to helping, let me know if you need anything here or on CNCF/K8s Slack.

Alex

howardjohn commented 3 years ago

Thanks, I will see if I can dig into this a bit.

Does Istio still require exec probes?

Nope, should no longer be needed

howardjohn commented 3 years ago

Ok, looks like this is what is needed to get latest Istio working. End result is everything is injected with mTLS, prometheus works, etc.

istioctl install -y

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
kubectl label namespace openfaas istio-injection=enabled --overwrite
kubectl label namespace openfaas-fn istio-injection=enabled --overwrite

# Enforce mTLS in openfaas namespaces
cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas-fn
spec:
  mtls:
    mode: STRICT
---
EOF

helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set functionNamespace=openfaas-fn \
    --set generateBasicAuth=true \
    --set gateway.directFunctions=true \ # REQUIRED!
    --set openfaasPRO=false
PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && \
echo "OpenFaaS admin password: $PASSWORD"

# Edit prometheus config kubernetes-pods:
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
# Replace all prometheus.io. with prometheus.io/ in all deployments
# Rename NATS port to "tcp" OR add "appProtocol: TCP" OR inject NATS. Or all 3

kubectl port-forward svc/gateway -n openfaas 31112:8080
# open ui, login with admin:$PASSWORD, deploy `env` function
curl http://127.0.0.1:31112/function/env

# Setup gateway, if you want. You could also use Ingress, but the one in helm chart is a deprecated extensions/v1beta1 version so I didn't want to use that
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: openfaas
  namespace: openfaas
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "openfaas/openfaas.example.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: openfaas
  namespace: openfaas
spec:
  hosts:
  - "openfaas.example.com"
  gateways:
  - openfaas/openfaas
  http:
  - route:
    - destination:
        host: gateway
EOF
curl <INGRESS_IP>/function/env -H "Host: openfaas.example.com"

I think the changes required (to nats service + prometheus config) should be compatible with non-istio as well.

alexellis commented 3 years ago

That was quick!

re: NATS

There are two ports which are both TCP, does renaming the name field to tcp help Istio / Envoy know what to do? For appProtocol - do you know what version is it compatible with? Is it K8s 1.18?

re: Prometheus

Can you explain the change? Backwards compatible sounds good. The way we used pod-based metrics was for custom metrics for HPAv2 https://docs.openfaas.com/tutorials/kubernetes-hpa-custom-metrics/

I tend to use arkade to add Ingress definitions, so it wouldn't surprise me about the chart ( which we should also update cc @LucasRoesler )

https://github.com/alexellis/arkade/blob/master/cmd/apps/openfaas_ingress_app.go

The Istio app for arkade uses a specific configuration to run in low-memory environments like Travis GitHub Action or in KinD. @stefanprodan put it together originally for Flagger, and we borrowed it for the app.

https://github.com/alexellis/arkade/blob/master/cmd/apps/istio_app.go

arkade can download a CLI like istioctl to do an installation now (that's how the linkerd app works). Are there some flags or tunings we could do in arkade install istio that would give a similar footprint?

Perhaps what Stefan has here? https://github.com/fluxcd/flagger/commit/a2774d92da8cea75ad1dc0615338cef28143c6a0#diff-e608f1fdaeb7eddba9483ae06ebbc5eccf7ceaa4c6c5044084ebe8d4cef1242eR15-R17

howardjohn commented 3 years ago

There are two ports which are both TCP, does renaming the name field to tcp help Istio / Envoy know what to do? For appProtocol - do you know what version is it compatible with? Is it K8s 1.18?

I think its GA in 1.20, so before then some clusters may have it turned off. The bummer is it will be rejected, rather than ignored, on old clusters. So probably not the best route for now. The naming is literally the name field. protocol refers to the L4 protocol, as a workaround for lack of appProtocol we give name meaning. See https://istio.io/latest/docs/ops/deployment/requirements/#server-first-protocols for more info, and specifically why NATS has this problem but almost all other applications don't (tl;dr, we cannot detect the protocol). Linkerd has the same problem - it looks like the current openfaas docs recommend disabling injection in NATS. I think you could enable it and add annotation: https://linkerd.io/2021/02/23/protocol-detection-and-opaque-ports-in-linkerd/, but I am not a linkerd expert.

Prometheus

This is a bit tricky. Basically, since we are enforcing mTLS, we have two ways to scrape metrics https://istio.io/latest/docs/ops/integrations/prometheus/#configuration.

The first is to modify the scraping config to use TLS. This is pretty istio specific so it may be a burden for openfaas. The other is rewriting the prometheus.io annotations to bypass mtls ports. The problem is this code looks for prometheus.io/{scrape,port,path} annotations. openfaas is using prometheus.io.scrape and prometheus.io.port (note . instead of /). In prom config, these actually end up identical due to escaping (__meta_kubernetes_pod_annotation_prometheus_io_path), but the Istio code doesn't detect the prometheus.io.scrape annotations so that is why it isn't automatically working. We should probably change istio to detect this as well, but I have never seen this before so I would recommend also switching to prometheus.io/ format - since the config in prometheus is identical I would be surprise if this broke anything.

The other option, and more broader, is changing how mTLS is configured. In the old days, Istio mTLS was all explicit config, which is why a flag like istio.mtls here makes sense. These days, its automatic - where possibly we will send mTLS. However, its not enforced. If we just don't apply any Istio mTLS config prometheus would also work without issues.

Ingress

I got my versions mixed up, its actually fine. But it will be removed in K8s 1.22 (in favor of networking/v1 api).

Istio install

Newer istio installs are slimmer out of the box now. You can also use helm (https://istio.io/latest/docs/setup/install/helm/). You can drop the resources:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      proxy:
        resources:
          requests:
            cpu: 10m
            memory: 40Mi
    pilot:
      resources:
        requests:
          cpu: 10m
          memory: 100Mi
    gateways:
      istio-ingressgateway:
        resources:
          requests:
            cpu: 10m
            memory: 100Mi

The same settings can be set through helm, just take the values: section and stick in some file and pass via helm install -f values.yaml

alexellis commented 3 years ago

Thanks @howardjohn we are taking another look at this today.

@LucasRoesler could you try this out please?

LucasRoesler commented 3 years ago

@howardjohn i am going to take a look at this tomorrow

LucasRoesler commented 3 years ago

@howardjohn i tried following the instructions

istioctl install -y

kubectl apply -f https://raw.githubusercontent.com/openfaas/faas-netes/master/namespaces.yml
kubectl label namespace openfaas istio-injection=enabled --overwrite
kubectl label namespace openfaas-fn istio-injection=enabled --overwrite

# Enforce mTLS in openfaas namespaces
cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: openfaas-fn
spec:
  mtls:
    mode: STRICT
---
EOF

helm upgrade openfaas --install openfaas/openfaas \
    --namespace openfaas  \
    --set functionNamespace=openfaas-fn \
    --set generateBasicAuth=true \
    --set gateway.directFunctions=true \ # REQUIRED!
    --set openfaasPRO=false
PASSWORD=$(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode) && \
echo "OpenFaaS admin password: $PASSWORD"

# Edit prometheus config kubernetes-pods:
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
# Replace all prometheus.io. with prometheus.io/ in all deployments
# Rename NATS port to "tcp" OR add "appProtocol: TCP" OR inject NATS. Or all 3

I did both of

Replace all prometheus.io. with prometheus.io/ in all deployments Rename NATS port to "tcp"

But i wasn't 100% sure what you mean by " Edit prometheus config kubernetes-pods:", so i edited the prometheus-config ConfigMap and then restarted prometheus.

But I can't get the gateway to start, clearly I missed something obvious

2021-04-03T09:34:33.285471Z error   failed scraping application metrics: error scraping http://localhost:8082/metrics: Get "http://localhost:8082/metrics": dial tcp [::1]:8082: connect: connection refused

When I use the same command to install OpenFaaS but without isitio configured, then it starts just fine.

howardjohn commented 3 years ago

failed scraping application metrics

Basically its looking for localhost:8082/metrics on the gateway, which is presumably configured with prometheus.io annotations for scraping? So not sure why its not working - can you check the gateway is actually serving metrics at that endpoint?

alexellis commented 3 years ago

I can run the following:

kubectl port-forward -n openfaas deploy/gateway 8082:8082 & 
curl localhost:8082/metrics

The K8s service for the gateway doesn't expose port 8082, but I'm not sure if that is relevant given that Prometheus is scraping the pod created by the deployment?

Edit: (without Istio)

howardjohn commented 3 years ago

I did the same steps and gateway was not working until I renamed the NATS port. Are you sure you did that before looking at the errors?

LucasRoesler commented 3 years ago

I am pretty sure i checked after the rename, but i will do it again to confirm and be sure

LucasRoesler commented 3 years ago

I just tried again, with the same results.

Here is the current state of the (crashing) gateway Pod

 kubectl describe po gateway-76c5df66dd-w6hrt
Name:         gateway-76c5df66dd-w6hrt
Namespace:    openfaas
Priority:     0
Node:         kind-control-plane/192.168.16.2
Start Time:   Wed, 07 Apr 2021 12:29:39 +0200
Labels:       app=gateway
              istio.io/rev=default
              pod-template-hash=76c5df66dd
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=gateway
              service.istio.io/canonical-revision=latest
Annotations:  prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istiod-ca-cert"],"ima...
Status:       Running
IP:           10.244.0.21
IPs:
  IP:           10.244.0.21
Controlled By:  ReplicaSet/gateway-76c5df66dd
Init Containers:
  istio-init:
    Container ID:  containerd://6c8d74cdecb1e7e19e427de2e6d2012e93e6bee8df89c7fa4656441572c58488
    Image:         docker.io/istio/proxyv2:1.9.2
    Image ID:      docker.io/istio/proxyv2@sha256:ee9c153e2f973937befb8af61e7269ab368020b6e7d91b5d891bbbeba55eb266
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x

      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 07 Apr 2021 12:29:48 +0200
      Finished:     Wed, 07 Apr 2021 12:29:48 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:        100m
      memory:     128Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from openfaas-controller-token-lkh7b (ro)
Containers:
  gateway:
    Container ID:   containerd://08078f4f99fc15522970c41cf2f848b4e99a453c2b5f26f6704ac90ca844d8b4
    Image:          ghcr.io/openfaas/gateway:0.20.12
    Image ID:       ghcr.io/openfaas/gateway@sha256:38a6c7a387921b76b09f11f72d44d98293203d920681deb1a35b92e336bdb040
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 07 Apr 2021 12:30:49 +0200
      Finished:     Wed, 07 Apr 2021 12:30:51 +0200
    Ready:          False
    Restart Count:  3
    Requests:
      cpu:      50m
      memory:   120Mi
    Liveness:   http-get http://:15020/app-health/gateway/livez delay=0s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get http://:15020/app-health/gateway/readyz delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      read_timeout:             65s
      write_timeout:            65s
      upstream_timeout:         60s
      functions_provider_url:   http://127.0.0.1:8081/
      direct_functions:         true
      direct_functions_suffix:  openfaas-fn.svc.cluster.local
      function_namespace:       openfaas-fn
      faas_nats_address:        nats.openfaas.svc.cluster.local
      faas_nats_port:           4222
      faas_nats_channel:        faas-request
      basic_auth:               true
      secret_mount_path:        /var/secrets
      auth_proxy_url:           http://basic-auth-plugin.openfaas:8080/validate
      auth_pass_body:           false
      scale_from_zero:          true
      max_idle_conns:           1024
      max_idle_conns_per_host:  1024
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from openfaas-controller-token-lkh7b (ro)
      /var/secrets from auth (ro)
  faas-netes:
    Container ID:   containerd://2d4354dfab3e745ec58fbc8900096697bd5902eb2119be86dbd8a363189bf02e
    Image:          ghcr.io/openfaas/faas-netes:0.13.2
    Image ID:       ghcr.io/openfaas/faas-netes@sha256:715342eda133a6296e8d9bdf89ec8e33d1c664ba5dd547b4f1207f7ee91b2e8c
    Port:           8081/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 07 Apr 2021 12:30:02 +0200
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:     50m
      memory:  120Mi
    Environment:
      port:                                   8081
      function_namespace:                     openfaas-fn
      read_timeout:                           60s
      profiles_namespace:                     openfaas
      write_timeout:                          60s
      image_pull_policy:                      Always
      http_probe:                             true
      set_nonroot_user:                       false
      readiness_probe_initial_delay_seconds:  2
      readiness_probe_timeout_seconds:        1
      readiness_probe_period_seconds:         2
      liveness_probe_initial_delay_seconds:   2
      liveness_probe_timeout_seconds:         1
      liveness_probe_period_seconds:          2
      cluster_role:                           false
    Mounts:
      /tmp from faas-netes-temp-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from openfaas-controller-token-lkh7b (ro)
  istio-proxy:
    Container ID:  containerd://69af614ca914ed2ec8f6dcc19aa81fface8e1e479c344be4feaf19943766d791
    Image:         docker.io/istio/proxyv2:1.9.2
    Image ID:      docker.io/istio/proxyv2@sha256:ee9c153e2f973937befb8af61e7269ab368020b6e7d91b5d891bbbeba55eb266
    Port:          15090/TCP
    Host Port:     0/TCP
    Args:
      proxy
      sidecar
      --domain
      $(POD_NAMESPACE).svc.cluster.local
      --serviceCluster
      gateway.$(POD_NAMESPACE)
      --proxyLogLevel=warning
      --proxyComponentLogLevel=misc:error
      --log_output_level=default:info
      --concurrency
      2
    State:          Running
      Started:      Wed, 07 Apr 2021 12:30:08 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   128Mi
    Readiness:  http-get http://:15021/healthz/ready delay=1s timeout=3s period=2s #success=1 #failure=30
    Environment:
      JWT_POLICY:                    first-party-jwt
      PILOT_CERT_PROVIDER:           istiod
      CA_ADDR:                       istiod.istio-system.svc:15012
      POD_NAME:                      gateway-76c5df66dd-w6hrt (v1:metadata.name)
      POD_NAMESPACE:                 openfaas (v1:metadata.namespace)
      INSTANCE_IP:                    (v1:status.podIP)
      SERVICE_ACCOUNT:                (v1:spec.serviceAccountName)
      HOST_IP:                        (v1:status.hostIP)
      CANONICAL_SERVICE:              (v1:metadata.labels['service.istio.io/canonical-name'])
      CANONICAL_REVISION:             (v1:metadata.labels['service.istio.io/canonical-revision'])
      PROXY_CONFIG:                  {}

      ISTIO_META_POD_PORTS:          [
                                         {"name":"http","containerPort":8080,"protocol":"TCP"}
                                         ,{"containerPort":8081,"protocol":"TCP"}
                                     ]
      ISTIO_META_APP_CONTAINERS:     gateway,faas-netes
      ISTIO_META_CLUSTER_ID:         Kubernetes
      ISTIO_META_INTERCEPTION_MODE:  REDIRECT
      ISTIO_METAJSON_ANNOTATIONS:    {"prometheus.io/port":"8082","prometheus.io/scrape":"true"}

      ISTIO_META_WORKLOAD_NAME:      gateway
      ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/openfaas/deployments/gateway
      ISTIO_META_MESH_ID:            cluster.local
      TRUST_DOMAIN:                  cluster.local
      ISTIO_PROMETHEUS_ANNOTATIONS:  {"scrape":"true","path":"","port":"8082"}
      ISTIO_KUBE_APP_PROBERS:        {"/app-health/gateway/livez":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":5},"/app-health/gateway/readyz":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"timeoutSeconds":5}}
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from openfaas-controller-token-lkh7b (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
      limits.cpu -> cpu-limit
      requests.cpu -> cpu-request
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  faas-netes-temp-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  auth:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  basic-auth
    Optional:    false
  openfaas-controller-token-lkh7b:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  openfaas-controller-token-lkh7b
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/arch=amd64
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  101s               default-scheduler  Successfully assigned openfaas/gateway-76c5df66dd-w6hrt to kind-control-plane
  Normal   Pulling    100s               kubelet            Pulling image "docker.io/istio/proxyv2:1.9.2"
  Normal   Pulled     92s                kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.2" in 7.730091949s
  Normal   Created    92s                kubelet            Created container istio-init
  Normal   Started    92s                kubelet            Started container istio-init
  Normal   Pulling    85s                kubelet            Pulling image "ghcr.io/openfaas/faas-netes:0.13.2"
  Normal   Pulled     85s                kubelet            Successfully pulled image "ghcr.io/openfaas/gateway:0.20.12" in 6.495277711s
  Normal   Created    79s                kubelet            Created container faas-netes
  Normal   Pulled     79s                kubelet            Successfully pulled image "ghcr.io/openfaas/faas-netes:0.13.2" in 6.691933971s
  Normal   Started    78s                kubelet            Started container faas-netes
  Normal   Pulling    78s                kubelet            Pulling image "docker.io/istio/proxyv2:1.9.2"
  Normal   Created    72s                kubelet            Created container istio-proxy
  Normal   Started    72s                kubelet            Started container istio-proxy
  Normal   Pulled     72s                kubelet            Successfully pulled image "docker.io/istio/proxyv2:1.9.2" in 6.774743084s
  Normal   Pulled     69s                kubelet            Successfully pulled image "ghcr.io/openfaas/gateway:0.20.12" in 2.025149046s
  Normal   Started    69s (x2 over 85s)  kubelet            Started container gateway
  Warning  Unhealthy  66s                kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff    66s                kubelet            Back-off restarting failed container
  Normal   Pulling    56s (x3 over 92s)  kubelet            Pulling image "ghcr.io/openfaas/gateway:0.20.12"
  Normal   Created    55s (x3 over 85s)  kubelet            Created container gateway
  Normal   Pulled     55s                kubelet            Successfully pulled image "ghcr.io/openfaas/gateway:0.20.12" in 1.453120997s

And here is the nats pod

# kubectl get po nats-c469b786c-gqr9z -oyaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/scrape: "false"
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2021-04-07T10:29:39Z"
  generateName: nats-c469b786c-
  labels:
    app: nats
    pod-template-hash: c469b786c
  name: nats-c469b786c-gqr9z
  namespace: openfaas
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: nats-c469b786c
    uid: 34519afd-6138-4dbf-83d1-2f089cd9a696
  resourceVersion: "3312"
  selfLink: /api/v1/namespaces/openfaas/pods/nats-c469b786c-gqr9z
  uid: 91d36c0d-6392-467c-b6a3-ec9e6d536439
spec:
  containers:
  - args:
    - --store
    - memory
    - --cluster_id
    - faas-cluster
    command:
    - /nats-streaming-server
    image: nats-streaming:0.17.0
    imagePullPolicy: Always
    name: nats
    ports:
    - containerPort: 4222
      name: tcp
      protocol: TCP
    resources:
      requests:
        memory: 120Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-blzk4
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: kind-control-plane
  nodeSelector:
    beta.kubernetes.io/arch: amd64
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: default-token-blzk4
    secret:
      defaultMode: 420
      secretName: default-token-blzk4
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2021-04-07T10:29:39Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2021-04-07T10:29:42Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2021-04-07T10:29:42Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2021-04-07T10:29:39Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://6e3df28f9f87f888475ce6d20ffd78ecee1c9193714c24528a49e70d5413ff9d
    image: docker.io/library/nats-streaming:0.17.0
    imageID: docker.io/library/nats-streaming@sha256:9d7edf7333ca43b1fdb30cd6290f4215d2be408f74edf6b829163c81cbb9d3d9
    lastState: {}
    name: nats
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2021-04-07T10:29:41Z"
  hostIP: 192.168.16.2
  phase: Running
  podIP: 10.244.0.18
  podIPs:
  - ip: 10.244.0.18
  qosClass: Burstable
  startTime: "2021-04-07T10:29:39Z"

howardjohn commented 3 years ago

Just to clarify, when I say rename the NATS port I mean the Service, not pod's port.

If that doesn't work, check out the gateway's logs in the gateway pod. In my case I saw it crashing due to failing to connect to NATS.

LucasRoesler commented 3 years ago

Ok, I had a change to retry today and ..... :muscle: success!

$ curl http://localhost:8080/function/env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=env-6dc77984d8-ssh48
fprocess=env
KUBERNETES_PORT=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
KUBERNETES_SERVICE_HOST=10.96.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
HOME=/home/app
Http_User_Agent=curl/7.68.0
Http_X_Forwarded_Host=localhost:8080
Http_Content_Length=0
Http_Accept=*/*
Http_X_Call_Id=d732b7a5-c1f0-443e-8596-efe2c87c9daa
Http_X_Start_Time=1618133183910952047
Http_Accept_Encoding=gzip
Http_X_Envoy_Attempt_Count=1
Http_X_B3_Spanid=2445da32d16249d7
Http_X_Forwarded_For=127.0.0.1:53382
Http_X_Forwarded_Proto=http
Http_X_B3_Traceid=6e374f868082ff89fd4d00fa8878c8b6
Http_X_Request_Id=71a8262b-3a08-4ed7-b2bf-600ed01ae7e6
Http_X_Forwarded_Client_Cert=By=spiffe://cluster.local/ns/openfaas-fn/sa/default;Hash=3575313a26a287d0f823273a79a383086d3d0518642008fb382166adfbf88c65;Subject="";URI=spiffe://cluster.local/ns/openfaas/sa/openfaas-controller
Http_X_B3_Parentspanid=fd4d00fa8878c8b6
Http_X_B3_Sampled=0
Http_Method=GET
Http_ContentLength=0
Http_Path=/
Http_Host=env.openfaas-fn.svc.cluster.local:8080

Renaming the port on the nats Service seems to be the missing piece. Once I did that, everything just started working as expected.

So the next step is to make the changes to the chart right?

alexellis commented 3 years ago

That's awesome 🥳

alexellis commented 3 years ago

@LucasRoesler was asking which changes are backwards compatible with different K8s versions?

We could use istio.mtls to indicate the newer, "breaking" changes and specify a different minimum Kubernetes version

howardjohn commented 3 years ago

None of this should be linked to Kubernetes versions. I would expect it to work on all k8s versions supported by Istio (1.15-1.20 at the time)

LucasRoesler commented 3 years ago

@howardjohn is there any chance you can provide more guidance on what to keep and what to remove from our configuration? I went to go update the chart today and realized how many more configurations there are already in the chart? Should those be kept or removed?

Also, faas-netes has a "mutli-namespace" mode, this is controlled via an annotation on the allowed target namespacs. How should these settings work if we need mTLS to several namespaces which are not known at installation? Are we going to need to bundle additional support in faas-netes or just provide additional documentation along the lines of "when using istion mTLS, in addition to the annoation on the namespace, you must install this additional PeerAuthentication"?

I am sorry this is taking so long. I am not an Istio user, so I am learning all of this as we go.

howardjohn commented 3 years ago

Also, faas-netes has a "mutli-namespace" mode, this is controlled via an annotation on the allowed target namespacs. How should these settings work if we need mTLS to several namespaces which are not known at installation? Are we going to need to bundle additional support in faas-netes or just provide additional documentation along the lines of "when using istion mTLS, in addition to the annoation on the namespace, you must install this additional PeerAuthentication"?

If you need to keep the istio.mtls mode for backwards compatibility, its fine. But I think it would be actually better to have users manage their own PeerAuthentications. Note that in modern Istio versions, traffic is ~always mTLS. PeerAuthentication just makes it so plaintext requests are not allowed. If you do keep it, you would probably need to document to install the additional PA in the extra namespaces, which is part of why I think we should just have the user manage it all together. They may even do it at the mesh-wide level which openfaas cannot/shouldn't really influence.

alexellis commented 3 years ago

We have a blog post for anyone landing here -> https://www.openfaas.com/blog/istio-functions/

Thanks to everyone who helped.

openfaas / faas-netes