search

solo-io / gloo

The Feature-rich, Kubernetes-native, Next-Generation API Gateway Built on Envoy
https://docs.solo.io/
Apache License 2.0
4.07k stars 435 forks source link

Custom external auth not working when Custom external auth server is different k8s cluster than Gloo Edge - Getting 404 for endpoint behind custom external auth #5581

Open rr-sarvesh-padia opened 2 years ago

rr-sarvesh-padia commented 2 years ago

Describe the bug I’m trying to setup custom ext-auth for Gloo Edge(open source) and facing some issues. Please find the details of my setup below -

Note - This issue is reproducible even with sample custom auth example on Gloo Edge's site, if we deploy custom auth server in different k8s cluster than Gloo Edge.

Note - Custom Authorization Server(http based) is in a different k8s cluster, I have created a static upstream to reach to that server. I have created a test VS which points to this upstream and made sure we can reach Authorization Server via test VS and static upstream. Settings CR - 

apiVersion: v1
items:
- apiVersion: gloo.solo.io/v1
  kind: Settings
  metadata:
    annotations:
      meta.helm.sh/release-name: gloo
      meta.helm.sh/release-namespace: gloo-system
    creationTimestamp: "2021-11-15T20:04:54Z"
    generation: 3
    labels:
      app: gloo
      app.kubernetes.io/managed-by: Helm
      gloo: settings
    name: default
    namespace: gloo-system
    resourceVersion: "1753388"
    selfLink: /apis/gloo.solo.io/v1/namespaces/gloo-system/settings/default
    uid: e7523607-62cd-44df-9d8d-4920e9c0f381
  spec:
    discovery:
      fdsMode: WHITELIST
    discoveryNamespace: gloo-system
    extauth:
      extauthzServerRef:
        name: authorization-server
        namespace: gloo-system
      httpService:
        request:
          allowedHeaders:
          - X-foo
      requestBody:
        maxRequestBytes: 10240
      requestTimeout: 0.5s
    gateway:
      readGatewaysFromAllNamespaces: false
      validation:
        allowWarnings: true
        alwaysAccept: true
        disableTransformationValidation: false
        proxyValidationServerAddr: gloo:9988
        validationServerGrpcMaxSizeBytes: 4000000
        warnRouteShortCircuiting: false
    gloo:
      disableKubernetesDestinations: false
      disableProxyGarbageCollection: false
      enableRestEds: false
      invalidConfigPolicy:
        invalidRouteResponseBody: Gloo Gateway has invalid configuration. Administrators
          should run `glooctl check` to find and fix config errors.
        invalidRouteResponseCode: 404
        replaceInvalidRoutes: false
      restXdsBindAddr: 0.0.0.0:9976
      xdsBindAddr: 0.0.0.0:9977
    kubernetesArtifactSource: {}
    kubernetesConfigSource: {}
    kubernetesSecretSource: {}
    refreshRate: 60s
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Upstreams CR -

apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gloo.solo.io/v1","kind":"Upstream","metadata":{"annotations":{},"name":"authorization-server","namespace":"gloo-system"},"spec":{"static":{"hosts":[{"addr":"authorization-service-qa1.coretech-np.rr-it.com","port":80}]}}}
  creationTimestamp: null
  generation: 2
  name: authorization-server
  namespace: gloo-system
  resourceVersion: "1715308"
spec:
  static:
    hosts:
    - addr: authorization-service-qa1.coretech-np.rr-it.com
      port: 80
status:
  statuses:
    gloo-system:
      reportedBy: gloo
      state: 1

Virtual service CR -

apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.solo.io/v1","kind":"VirtualService","metadata":{"annotations":{},"name":"auth-service","namespace":"gloo-system"},"spec":{"virtualHost":{"domains":["*.coretech-np.rr-it.com"],"options":{"extauth":{"customAuth":{}}},"routes":[{"matchers":[{"prefix":"/auth-int"}],"routeAction":{"single":{"upstream":{"name":"apigw-rrus-auth-80","namespace":"gloo-system"}}}},{"matchers":[{"prefix":"/auth"}],"routeAction":{"single":{"upstream":{"name":"apigw-rrus-auth-80","namespace":"gloo-system"}}}},{"matchers":[{"prefix":"/version"}],"routeAction":{"single":{"upstream":{"name":"apigw-rrus-auth-80","namespace":"gloo-system"}}}}]}}}
  creationTimestamp: "2021-11-15T20:20:01Z"
  generation: 12
  name: auth-service
  namespace: gloo-system
  resourceVersion: "1826297"
  selfLink: /apis/gateway.solo.io/v1/namespaces/gloo-system/virtualservices/auth-service
  uid: 84adda1c-141d-4179-bf58-52afb9afda28
spec:
  virtualHost:
    domains:
    - '*.coretech-np.rr-it.com'
    options:
      extauth:
        customAuth: {}
    routes:
    - matchers:
      - prefix: /version
      routeAction:
        single:
          upstream:
            name: apigw-rrus-auth-80
            namespace: gloo-system
status:
  statuses:
    gloo-system:
      reportedBy: gateway
      state: 1
      subresourceStatuses:
        '*v1.Proxy.gloo-system.gateway-proxy':
          reportedBy: gloo
          state: 1

I’m getting 404 when I call an endpoint which is behind external custom auth in my Virtual service.

To Reproduce Steps to reproduce the behavior:

  1. Install Gloo edge open source version - helm install gloo . --namespace gloo-system --create-namespace -f value-overrides.yaml

value-overrides.yaml 

global:
  extensions:
    extAuth:
      extauthzServerRef:
        name: authorization-server
        namespace: gloo-system
      httpService:
        pathPrefix: "/authorization/v1/authorize"
        request:
          allowedHeaders:
          - "X-foo"
      requestBody:
        maxRequestBytes: 10240
      requestTimeout: 0.5s

  1. create static upstream to connect to external custom auth server kubectl apply --filename authorization-upstream.yaml authorization-upstream.yaml

    apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
name: authorization-server
namespace: gloo-system
spec:
static:
hosts:
- addr: authorization-service-qa1.coretech-np.rr-it.com
  port: 80

  2. create virtual service kubectl apply --filename authentication-service-vs.yaml

    apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: auth-service
namespace: gloo-system
spec:
virtualHost:
domains:
- '*.coretech-np.rr-it.com'
routes:
- matchers:
  - prefix: /version
  routeAction:
    single:
      upstream:
        name: apigw-rrus-auth-80
        namespace: gloo-system

  3. Verify you can reach the service -

    US06321947/Users/Shared/workspace/gloo(feature/use-value-override-file|✚1…) % curl -H "Host: gloo-poc.coretech-np.rr-it.com" $(glooctl proxy url)/version
{"LastCommit":"b5df8dd9ad0a2cab1b08983373d9e15bcccb9d97","Branch":"feature/apigw"}%

  4. Make changes in VS to add external custom auth - kubectl apply --filename authentication-service-vs.yaml

    apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: auth-service
namespace: gloo-system
spec:
virtualHost:
domains:
- '*.coretech-np.rr-it.com'
options:
  extauth:
    customAuth: {}
routes:
- matchers:
  - prefix: /version
  routeAction:
    single:
      upstream:
        name: apigw-rrus-auth-80
        namespace: gloo-system

  5. Check if you can reach application endpoint behind external custom auth -

    curl -H "Host: gloo-poc.coretech-np.rr-it.com" $(glooctl proxy url)/version
<span>The page you're looking for could not be found.</span>%

Expected behavior Should be able to reach the endpoint behind external custom auth if auth passes and should not see 404.

Additional context Add any other context about the problem here, e.g.

rr-sarvesh-padia commented 2 years ago

Relevant ExtAuth received from glooctl proxy dump -n gloo-system command

{
    "name": "envoy.filters.http.ext_authz",
    "typed_config":
    {
        "@type": "type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz",
        "http_service":
        {
            "server_uri":
            {
                "uri": "http://not-used.example.com/",
                "cluster": "auth-server_gloo-system",
                "timeout": "0.500s"
            },
            "authorization_request":
            {
                "allowed_headers":
                {
                    "patterns":
                    [
                        {
                            "exact": "X-foo"
                        }
                    ]
                }
            }
        },
        "with_request_body":
        {
            "max_request_bytes": 10240
        },
        "metadata_context_namespaces":
        [
            "envoy.filters.http.jwt_authn"
        ],
        "transport_api_version": "V3"
    }
}

Please note - typed_config.http_service.server_uri is not set. Most probably this is root cause of this issue.

chrisgaun commented 2 years ago

It might be easier to get help on questions in the community Slack https://slack.solo.io/

rr-sarvesh-padia commented 2 years ago

It might be easier to get help on questions in the community Slack https://slack.solo.io/

I asked question related to this in the community Slack. They asked me to create an GH issue. Thanks https://solo-io.slack.com/archives/C9L6VPAUW/p1637020406078600

github-actions[bot] commented 3 months ago

This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.