External authentication does not work for me on local cluster

[Cito]

Describe the bug

Authentication using an external authentication service does not seem to work on a Kubernetes Cluster running on Docker Desktop for Mac. The authentication service is never accessed, and the authentication fails with error code 403 (forbidden).

I was able to reproduce the same problem with Docker Desktop for Windows, and Minikube on Mac (commands to reproduce only shown for Docker Desktop for Mac, adapt accordingly).

To Reproduce

• Install Docker Desktop for Mac, Enable Kubernetes.

• Install Emissary-Ingress 3.0.0 and demo app according to the docs here:

kubectl create namespace emissary && \
kubectl apply -f https://app.getambassador.io/yaml/emissary/3.0.0/emissary-crds.yaml && \
kubectl wait --timeout=90s --for=condition=available deployment emissary-apiext -n emissary-system
kubectl apply -f https://app.getambassador.io/yaml/emissary/3.0.0/emissary-emissaryns.yaml && \
kubectl -n emissary wait --for condition=available --timeout=90s deploy -lproduct=aes

kubectl apply -f - <<EOF
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-listener-8080
  namespace: emissary
spec:
  port: 8080
  protocol: HTTP
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL
EOF

kubectl apply -f https://app.getambassador.io/yaml/v2-docs/3.0.0/quickstart/qotm.yaml

kubectl apply -f - <<EOF
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: quote-backend
spec:
  hostname: "*"
  prefix: /backend/
  service: quote
  docs:
    path: "/.ambassador-internal/openapi-docs"
EOF

• Confirm that this works:

  curl -i http://localhost/backend/

• Should output something like this:

  HTTP/1.1 200 OK
  ...
  {
  "quote": "Nihilism gambles with lives, happiness, and even destiny itself!",
  ...
  }

• Install basic authentication demo according to the docs here:

kubectl apply -f https://app.getambassador.io/yaml/v2-docs/3.0.0/demo/demo-auth.yaml

kubectl apply -f https://app.getambassador.io/yaml/v2-docs/3.0.0/demo/demo-auth-enable.yaml

• Check again if this works:

  curl -i -u username:password http://localhost/backend/get-quote/

As a result, I get an error 403:

HTTP/1.1 403 Forbidden

When examining the logs as explained here, I find the following messages:

[source/extensions/filters/http/ext_authz/ext_authz.cc:75] [C1109][...] ext_authz filter calling authorization server
[source/common/router/router.cc:470] [C0][...] cluster 'cluster_extauth_example_auth_3000_default' match for URL '/extauth/backend/'
[source/common/upstream/cluster_manager_impl.cc:1632] no healthy host for HTTP connection pool
[source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=false):
':status', '503'
'content-length', '19'
'content-type', 'text/plain'

Also, the authentication service does not seem to be accessed by Emissary-ingress.

Expected behavior

The authentication service should be accessed, and authentication should be successful.

This should give a similar output as without authentication:

HTTP/1.1 200 OK
...
{
    "quote": "Nihilism gambles with lives, happiness, and even destiny itself!",
    ...
}

Even with wrong credentials, it should give an error 401, not 403.

Versions (please complete the following information):

• Emissary-ingress: 3.0.0
• Kubernetes environment: Docker Desktop for Mac
• Docker for Desktop Version 4.11.0
• Kubernetes Version 1.24.2

[Cito]

After investigating this today a bit more I think I found the cause of the problem: Emissary-ingress is looking for the AuthService in the "emissary" namespace, not in the "default" namespace, where it was created.

So the solution is to either create the AuthService in the "emissary" namespace, or to specify the namespace explicitly in the auth_service field.

In the config file https://app.getambassador.io/yaml/v2-docs/3.0.0/demo/demo-auth-enable.yaml it is currently specified like this:

apiVersion: getambassador.io/v2
kind:  AuthService
name:  authentication
auth_service: "example-auth:3000"
path_prefix: "/extauth"
allowed_request_headers:
- "x-qotm-session"
allowed_authorization_headers:
- "x-qotm-session"

After making the following change, everything worked like expected:

auth_service: "example-auth.default:3000"

Leaving this open, maybe someone can confirm my analysis and fix the documentation, and maybe add some remarks about namespaces.

[cindymullins-dw]

Thanks, @Cito for this catch! I've created a docs ticket for this.