search

lcc3108 / lcc3108.github.io

https://lcc3108.github.io
MIT License
0 stars 0 forks source link

Istio+Dex(OIDC)를 이용한 HTTP 서비스 인증 | 너구리는 라쿤이 아니다 #3

Open utterances-bot opened 3 years ago

utterances-bot commented 3 years ago

Istio+Dex(OIDC)를 이용한 HTTP 서비스 인증 | 너구리는 라쿤이 아니다

사전지식 OIDC란 idP Dex 사전 준비사항 환경설정 kubectl alias k Github OAuth 앱 등록 Git Clone해서 하기 CLI로 하기 kubernet...

https://lcc3108.github.io/articles/2020-12/Istio+Dex-%EC%9D%B8%EC%A6%9D

amalendur commented 3 years ago

Hi, Thank You for such a wonderful documentation. In our environment we are trying to achieve the same thing but without any luck.

We are deployed Istio on the top of AWS EKS cluster. Deploy all the cluster components through helm(using template file).

We are using following versions;

EKS Version : 1.20
Helm Version: 3.5.3
Kubectl Version: 1.21.0(client)/1.20.7(server)
Istio Version: 1.11.1

dex config template#

grpc: false
certs:
  grpc:
    create: false
  web:
    create: false
ports:
  web:
    servicePort: 5556
  telemetry:
    servicePort: 5558
config:
  issuer: https://dex.example.com
  connectors:
  - type: mockCallback
    id: mock
    name: <cluster_name>
  enablePasswordDB: true
  staticClients:
  - id: 'oidc-auth-client'
    redirectURIs:
    - "https://dashboard.example.com/oauth2/callback"
    name: "oidc-auth-client"
    secret: KYCI4XWYZGhu8cAC0hq0xtf1XWcQxPBU1HyzOZdGxi8=
  staticPasswords:
  - email: "admin@example.com"
    hash: "$2a$10$2b2cU8CPhOTaGrs1HRQuAueS7JTT5ZHsHSzYiFPm1leZck7Mc8T4W"
    username: "admin"
    userID: "08a8684b-db88-4b73-90a9-3cd1661f5466"
  telemetry:
    http: 0.0.0.0:5558
  frontend:
    theme: coreos
ingress:
  enabled: false
service:
  type: ClusterIP
livenessProbe:
  enabled: true
  initialDelaySeconds: 1
  failureThreshold: 1
  httpPath: "/healthz"
  periodSeconds: 10
  timeoutSeconds: 5

oath2-proxy config template#

service:
  type: ClusterIP
  port: 4180
  #annotations:
    ##To ensure VirtualService that cluster have the external fqdn in the hosts section
    #ingress.kubernetes.io/service-upstream: "true"
config:
  clientID: 'oidc-auth-client'
  clientSecret: KYCI4XWYZGhu8cAC0hq0xtf1XWcQxPBU1HyzOZdGxi8=
  cookieSecret: KYCI4XWYZGhu8cAC0hq0xtf1XWcQxPBU1HyzOZdGxi8=
extraArgs:
  provider: oidc
  provider-display-name: dex
  proxy-websockets: true
  oidc-issuer-url: https://dex.example.com
  cookie-secure: true
  cookie-name: auth
  cookie-refresh: 1h
  cookie-expire: 4h
  cookie-httponly: true
  email-domain: "*"
  pass-host-header: true
  ping-path: /ping
  set-authorization-header: true
  skip-provider-button: true
  http-address: 0.0.0.0:4180
  upstream: static://200
  scope: openid profile email offline_access groups
  client-secret: KYCI4XWYZGhu8cAC0hq0xtf1XWcQxPBU1HyzOZdGxi8=
  client-id: 'oidc-auth-client'
  whitelist-domain: .example.com
  cookie-domain: .example.com
ingress:
  enabled: false

the envoyfilter initially I have tried with as following;

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: kubernetes-dashboard
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
              subFilter:
                name: istio.metadata_exchange
            sni: dashboard.example.com
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.ext_authz
          typed_config:
            "@type": "type.googleapis.com/envoy.config.filter.http.ext_authz.v2.ExtAuthz"
            http_service:
              server_uri:
                uri: "http://oauth2-proxy.auth.svc.cluster.local/"
                timeout: 1.5s
                cluster: outbound|4180||oauth2-proxy.auth.svc.cluster.local
              authorizationRequest:
                allowedHeaders:
                  patterns:
                    - exact: "cookie"
                    - exact: "authorization"
              authorizationResponse:
                allowedClientHeaders:
                  patterns:
                    - exact: "set-cookie"
                    - exact: "authorization"
                allowedUpstreamHeaders:
                  patterns:
                    - exact: "set-cookie"
                    - exact: "authorization"

the 2nd envoyfilter IO have tried with;

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: kubernetes-dashboard
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: istio.metadata_exchange
          sni: dashboard.example.com
    patch:
      operation: INSERT_AFTER
      value:
        name: envoy.filters.http.ext_authz
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          http_service:
            authorizationRequest:
              allowedHeaders:
                patterns:
                - exact: accept
                - exact: authorization
                - exact: cookie
                - exact: from
                - exact: proxy-authorization
                - exact: user-agent
                - exact: x-forwarded-access-token
                - exact: x-forwarded-email
                - exact: x-forwarded-for
                - exact: x-forwarded-host
                - exact: x-forwarded-proto
                - exact: x-forwarded-user
                - prefix: x-auth-request
                - prefix: x-forwarded
            authorizationResponse:
              allowedClientHeaders:
                patterns:
                - exact: authorization
                - exact: location
                - exact: proxy-authenticate
                - exact: set-cookie
                - exact: www-authenticate
                - prefix: x-auth-request
                - prefix: x-forwarded
              allowedUpstreamHeaders:
                patterns:
                - exact: authorization
                - exact: location
                - exact: proxy-authenticate
                - exact: set-cookie
                - exact: www-authenticate
                - prefix: x-auth-request
                - prefix: x-forwarded
            server_uri:
              cluster: outbound|4180||oauth2-proxy.auth.svc.cluster.local
              timeout: 1.5s
              uri: http://oauth2-proxy.auth.svc.cluster.local

Nothing working. When we trying to access k8s-dashboard (https://dashboard.example.com) it's redirecting to "https://dashboard-ar-usv-poc-plt.devops.ada.iqvia.com/#/overview?namespace=default". there was no log generated in oauth2-proxy. I found some log in istio-ingressgateway as following; {"level":"debug","time":"2021-10-26T07:58:22.579160Z","scope":"envoy router","msg":"[C117][S17481825958129025397] cluster 'outbound|80||kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local' match for URL '/api/v1/namespace'"}

Highly appreciate your help :)

amalendur commented 3 years ago

correction :) 2nd Envoyfilter I have tried with as following (without any luck!)

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: kubernetes-dashboard
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: istio.metadata_exchange
          sni: dashboard.example.com
    patch:
      operation: INSERT_AFTER
      value:
        name: envoy.filters.http.ext_authz
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          http_service:
            authorizationRequest:
              allowedHeaders:
                patterns:
                - exact: accept
                - exact: authorization
                - exact: cookie
                - exact: from
                - exact: proxy-authorization
                - exact: user-agent
                - exact: x-forwarded-access-token
                - exact: x-forwarded-email
                - exact: x-forwarded-for
                - exact: x-forwarded-host
                - exact: x-forwarded-proto
                - exact: x-forwarded-user
                - prefix: x-auth-request
                - prefix: x-forwarded
            authorizationResponse:
              allowedClientHeaders:
                patterns:
                - exact: authorization
                - exact: location
                - exact: proxy-authenticate
                - exact: set-cookie
                - exact: www-authenticate
                - prefix: x-auth-request
                - prefix: x-forwarded
              allowedUpstreamHeaders:
                patterns:
                - exact: authorization
                - exact: location
                - exact: proxy-authenticate
                - exact: set-cookie
                - exact: www-authenticate
                - prefix: x-auth-request
                - prefix: x-forwarded
            server_uri:
              cluster: outbound|4180||oauth2-proxy.auth.svc.cluster.local
              timeout: 1.5s
              uri: http://oauth2-proxy.auth.svc.cluster.local
amalendur commented 3 years ago

Gateway with wildcard domain;

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
    #istio: ingressgateway
  servers:
  - hosts:
    - "*.example.com"
    port:
      name: https
      number: 443
      protocol: HTTP

Virtual services;

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: dex
  namespace: auth
spec:
  gateways:
  - istio-system/gateway
  hosts:
  - dex.example.com
  http:
  - route:
    - destination:
        host: dex
        port:
          number: 5556
    retries:
      attempts: 3
      perTryTimeout: 5s
      retryOn: gateway-error,connect-failure,refused-stream

---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: oauth2-proxy
  namespace: auth
spec:
  gateways:
  - istio-system/gateway
  hosts:
  - auth.example.com
  http:
  - route:
    - destination:
        host: oauth2-proxy
        port:
          number: 4180
    retries:
      attempts: 3
      perTryTimeout: 5s
      retryOn: gateway-error,connect-failure,refused-stream

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  gateways:
    - istio-system/gateway
  hosts:
  - dashboard.example.com
  http:
  - route:
    - destination:
        host: kubernetes-dashboard
        port:
          number: 80
    retries:
      attempts: 3
      perTryTimeout: 5s
      retryOn: gateway-error,connect-failure,refused-stream
amalendur commented 3 years ago

namespaces (auth, kubernetes-dashboard) are labeled with "istio-injection=enabled".

lcc3108 commented 3 years ago

@amalendur The answer was slow because I was busy. In my opinion, the envoy filter matches based on SNI, but it seems to be a problem because the gateway uses HTTP protocol.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: kubernetes-dashboard
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
            subFilter:
              name: istio.metadata_exchange
          sni: dashboard.example.com # <- use sni match on tls or https
    patch:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    app: istio-ingressgateway
    #istio: ingressgateway
  servers:
  - hosts:
    - "*.example.com"
    port:
      name: https
      number: 443
      protocol: HTTP <- gateway protocol use http, and http has no SNI
lcc3108 commented 3 years ago

In our environment, we are deploying Istio on top of AWS EKS cluster and we are using AWS_ACM certificate (on a single wildcard domain [*.example.com]). istio-ingressgateway configure by using that ACM certificate.

istio-ingressgateway:
    type: LoadBalancer
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: <acm_arn>
    externalTrafficPolicy: "Local"

I have tried with following gateway config;

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "*"
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - "*.example.com"
    port:
      name: https
      number: 443
      protocol: HTTP

Unfortunately there is no luck.

Would you mind if I ask about your test cluster? Was it on EKS?

@amalendur

I copied it for others in the future with similar questions to see.

Yes, we are using eks,istio, and nlb all. The difference is that I use certmanager without authentication using acm.(certmanager, acm auto intergration not free) In addition, tls termination is configured to be delivered at the station gateway, not at nlb.

Untitled (1)

In your case, ingress gateway cannot receive snihost because tls termination occurs with the certificate of acm in nlb. (As far as I know)

You can only receive the Host header as shown in the picture above.