Kubeflow with OIDC possible scopes error

[GdMacmillan]

Hi, for context, I work for a DS team within large organization. Myself and a DevOps team member are attempting to deploy Kubeflow which must conform to security requirements. We were able to provision the cluster and node groups correctly following the documents here and also employ a config map to maintain instance security policies.

The issue we seemed to have run across applies to the auth configuration. The config file we are using is here: https://raw.githubusercontent.com/kubeflow/manifests/v1.0-branch/kfdef/kfctl_aws_cognito.v1.0.2.yaml

Auth is configured to use OIDC using the instructions [here] (https://www.kubeflow.org/docs/aws/authentication-oidc/) with github providing user federation as we cannot use Cognito due to corporate policy. We get errors in the authzadapter seemingly related to scopes: kubectl logs authzadaptor-89c9d49bc-dbnr9 -n istio-system

time="2020-06-01T21:04:21Z" level=error msg="Email doesn't exist in user's claim map[exp:1.591045581e+09 iss:https://vlt-kubeflow.auth0.com/ name:Mark Persiko nickname:mcpersiko picture:https://avatars3.githubusercontent.com/u/32829230?v=4 sub:github|32829230 updated_at:2020-06-01T21:04:20.541Z]. This is not supported"
time="2020-06-01T21:04:21Z" level=error msg="Email doesn't exist in user's claim map[exp:1.591045581e+09 iss:https://vlt-kubeflow.auth0.com/ name:Mark Persiko nickname:mcpersiko picture:https://avatars3.githubusercontent.com/u/32829230?v=4 sub:github|32829230 updated_at:2020-06-01T21:04:20.541Z]. This is not supported"

We tried modifying the oidc-scope's to 'openid profile'. We also had to add at least two public subnets as an annotation (Istio can't seem to find these on its own.):

C02Z72G8LVDR:oidc mark$ cat ingress.yaml
apiVersion: extensions/v1beta1 # networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: istio-ingress
  annotations:
    alb.ingress.kubernetes.io/auth-type: oidc
    alb.ingress.kubernetes.io/auth-idp-oidc: '{"Issuer":"$(oidcIssuer)","AuthorizationEndpoint":"$(oidcAuthorizationEndpoint)","TokenEndpoint":"$(oidcTokenEndpoint)","UserInfoEndpoint":"$(oidcUserInfoEndpoint)","SecretName":"$(oidcSecretName)"}'
    alb.ingress.kubernetes.io/certificate-arn: $(certArn)
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/auth-scope: 'openid profile'
    alb.ingress.kubernetes.io/subnets: 'subnet-###############, subnet-###############'

You can see where we had to manually add the subnets for the ALB.

The manifests file we use is https://github.com/kubeflow/manifests/archive/v1.0-branch.tar.gz. To modify we download and unzip the manifest: tar -xvxf <manifest-file> cd to <manifest-dir>/aws/istio-ingress/overlays/oidc and edited the file ingress.yaml in a text editor. We package the manifest file back up with tar -cvzf <manifest-file> <manifest-dir> and edit the line in the kfctl config to use this file.

Removing auth, we can access the kubeflow dashboard and ingress is working.

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/bug	0.56

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[yanniszark]

cc @Jeffwan /platform aws

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
platform/aws	0.53

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[Jeffwan]

/assign @Jeffwan

[Jeffwan]

alb.ingress.kubernetes.io/auth-scope: 'openid profile'

See the manifest, we assume to use email here, if you don't have email. that would be a problem. @GdMacmillan https://github.com/kubeflow/manifests/blob/316c3f16698fce5de8349daf1a59756aeaeec18a/aws/istio-ingress/overlays/oidc/ingress.yaml#L10

[GdMacmillan]

Thanks @Jeffwan we're taking a look at it...