Service account does not have permission to access the IAP-protected application

[jal06]

This issue is a following of issue #https://github.com/kubeflow/kfserving/issues/1199 from KFServing I deployed Kubeflow 1.1 on GCP, and also installed the istio local gateway as described in #https://github.com/kubeflow/gcp-blueprints/pull/177

I would like to use KFServing and followed the instructions from the sample gcp_iap described here
I successfully deployed the inference service sklearn-iap-no-authz.yaml and successfully sent query to this inference service Then, I successfully deployed the inference service sklearn-iap-with-authz.yaml. However I get an error "Service account does not have permission to access the IAP-protected application (error 403 raised in iap_request.py )

Before sending request to inferenceservice, I followed the documentation https://cloud.google.com/iap/docs/authentication-howto and it looks working regarding IAP auth. Below are the steps I did

1. I created a client ID (Application type). I named it kfserving, and I got a client id and a client secret
2. I signed in to the application and I got an authorization code using the client id received above
3. I got the login token (refresh_token) using the authorization code, the client id and the client secret received above
4. I got a JSON object with an id_token using the refresh_token, the client id and the client secret received above
5. I was able to access the app using the id_token (return code 302 see additional details below)
6. I try to send a query request using make-prediction.sh , and I get the error Service account <kfname>-user@<project_id>.iam.gserviceaccount.com does not have permission to access the IAP-protected application. The service account I'd like to use is the sa which is automatically created when deploying Kubeflow, with a name like <kfname>-user@<project_id>.iam.gserviceaccount.com This service account has the roles "Editor" , "IAP-secured Web App User" and "Viewer" for the resources described in the Cloud IAP page (ressources kube-system/default-http-backend and istio-system/istio-ingressgateway )

I don't know what is wrong. Any help would be appreciated.

Below are some additional details : In step 4 (getting the id_token), I set IAP_CLIENT_ID (parameter --data audience) using the client I created when setting up OAuth for Cloud IAP before deploying Kubeflow as described here . Please, could you confirm that this is the right client_id ? In step 5, when using the command curl --verbose --header 'Authorization: Bearer ID_TOKEN' URL, I received a return code 302

 Connected to <kfname>.endpoints.<project_id>.cloud.goog (<ip_addesss>) port 80 (#0)
> GET / HTTP/1.1
> Host: <kfname>.endpoints.<project_id>.cloud.goog
> User-Agent: curl/7.71.1
> Accept: */*
> Authorization: Bearer blablabla...blablabla> 
 Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: https://<kfname>.endpoints.<project_id>.cloud.goog/
< X-Goog-IAP-Generated-Response: true
< Content-Length: 21
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 01 Dec 2020 15:28:25 GMT
< 
 Connection #0 to host <kfname>.endpoints.<project_id>.cloud.goog left intact
Redirecting to HTTPS.

In step 6, I set the environment as below in make-prediction.sh :

export GOOGLE_APPLICATION_CREDENTIALS=/home/jal/.config/gcloud/<kfname>-user-credentials.json

# Set the environment
INFERENCE_SERVICE=sklearn-iap
INPUT_PATH=./iris-input.json
PROJECT='<project_id>'
NAMESPACE='jal'
INGRESS_DNS=<kfname>.endpoints.<project_id>.cloud.goog
IAP_CLIENT_ID='<client_id>'
SERVICE_URL=https://${INGRESS_DNS}/kfserving/${NAMESPACE}/${INFERENCE_SERVICE}:predict

[Bobgy]

Thanks for the detailed instructions @jal06!

I'm not 100% sure, but I guess the problem is after step 4.

1. When you use service account for authentication, you should follow https://cloud.google.com/iap/docs/authentication-howto#authenticating_from_a_service_account

2. I set IAP_CLIENT_ID (parameter --data audience) using the client I created when setting up OAuth for Cloud IAP before deploying Kubeflow as described here .

This is incorrect. You should create another OAuth client of desktop application type. The client you set up for Cloud IAP during the deployment process is meant to only used by Kubeflow to verify incoming requests from IAP.

[Bobgy]

I think you can refer to documentation for connecting to Kubeflow Pipelines protected by IAP: https://www.kubeflow.org/docs/gke/pipelines/authentication-sdk/#connecting-to-kubeflow-pipelines-in-a-full-kubeflow-deployment.

It should be similar.

[Bobgy]

This can likely be caused by https://github.com/kubeflow/gcp-blueprints/pull/177#issuecomment-756124781. I should try to figure out a solution to https://github.com/kubeflow/gcp-blueprints/issues/176 first, then come to this one