MarkEdmondson1234 / googleCloudRunner

Easy R scripts on Google Cloud Platform via Cloud Run, Cloud Build and Cloud Scheduler
https://code.markedmondson.me/googleCloudRunner/
Other
81 stars 26 forks source link

Unable to run Cloud Build - Docker test #96

Closed smach closed 2 years ago

smach commented 3 years ago

As we discussed on Slack, here's the output when I try the cr_setup_test()

cr_setup_test() i Perform deployments to test your setup is working. Takes around 5mins. ESC or 0 to skip. Select which deployments to test

1: All tests 2: Cloud Build - Docker 3: Cloud Run - plumber API with Pub/Sub 4: Cloud Build - R script 5: Cloud Scheduler - R script

Selection: 2

i Attempting Docker deployment on Cloud Build via cr_deploy_docker() i 2020-12-21 15:11:48 > Building C:/Users/Sharon/Documents/R/win-library/4.0/googleCloudRunner/example folder for Docker image: example i 2020-12-21 15:11:48 > Configuring Dockerfile i 2020-12-21 15:11:48 > Dockerfile found in C:/Users/Sharon/Documents/R/win-library/4.0/googleCloudRunner/example i 2020-12-21 15:11:48 > Image to be built: gcr.io/MYPROJECT/example

-- #Deploy docker build for image: gcr.io/MYPROJECT/e

-- #Upload C:/Users/Sharon/Documents/R/win-library/4.0/googleC i 2020-12-21 15:11:48 > Copying files from C:/Users/Sharon/Documents/R/win-library/4.0/googleCloudRunner/example to /deploy i 2020-12-21 15:11:48 > Compressing files from /deploy to example.tar.gz i 2020-12-21 15:11:48 > Uploading example.tar.gz to MYSTORAGEBUCKET/example.tar.gz i 2020-12-21 15:11:48 > Set API cache 2020-12-21 15:11:48 -- File size detected as 846 bytes 2020-12-21 15:11:48 -- Simple upload i 2020-12-21 15:11:48 > Request: https://www.googleapis.com/upload/storage/v1/b/MYSTORAGEBUCKET/o/?uploadType=media&name=example.tar.gz&predefinedAcl=bucketOwnerFullControl i 2020-12-21 15:11:48 > Could not parse body JSON i 2020-12-21 15:11:48 > Request: https://cloudbuild.googleapis.com/v1/projects/MYPROJECT/builds/ i 2020-12-21 15:11:48 > Body JSON parsed to: {"steps":[{"name":"gcr.io/kaniko-project/executor:latest","args":["-f","Dockerfile","--destination","gcr.io/MYPROJECT/example:latest","--context=dir:///workspace/deploy/example","--cache=true"],"dir":"deploy/example","waitFor":["-"]},{"name":"gcr.io/kaniko-project/executor:latest","args":["-f","Dockerfile","--destination","gcr.io/MYPROJECT/example:$BUILD_ID","--context=dir:///workspace/deploy/example","--cache=true"],"dir":"deploy/example","waitFor":["-"]}],"source":{"storageSource":{"bucket":"MYBUCKET","object":"example.tar.gz"}},"timeout":"600s"} i 2020-12-21 15:11:49 > Request Status Code: 403 i 2020-12-21 15:11:49 > API returned error: The caller does not have permission i 2020-12-21 15:11:49 > No retry attempted: The caller does not have permission Error: API returned: The caller does not have permission

MarkEdmondson1234 commented 3 years ago

Thanks, it does seem that the call to Cloud Build itself is unauthorised, which is odd if you auto-authenticate on package load. Doing it the old fashioned way may I ask you to check the following

  1. Do you get the "Successfully auto-authenticated via XXXX" pointing at your service key json file on package load?
  2. Does Sys.getenv("GCE_AUTH_FILE") resolve to the right place (same as 1) and if you issue it before cr_setup_test() is it still set to the same?
  3. In the GCP console "IAM & Admin" section do you see that service email name (something like googlecloudrunner@project-name.iam.gserviceaccount.com) with these permissions:
Screenshot 2020-12-22 at 00 07 09
  1. Is it a new or old Google Cloud Platform project?
  2. Is the Cloud Build API activated? (https://console.cloud.google.com/apis/api/cloudbuild.googleapis.com/overview)

I'd also like to figure out why this didn't happen during cr_setup(), perhaps restarting that may help.

smach commented 3 years ago
  1. Yes Setting scopes to https://www.googleapis.com/auth/cloud-platform Successfully auto-authenticated via D:/dockerapps/MYKEY.json

  2. Yes

    Sys.getenv("GCE_AUTH_FILE") [1] "D:/dockerapps/MYKEY.json"

  3. Yes

  4. It's an old project

  5. Yes (since Cloud Build API has an option to "Disable API")

MarkEdmondson1234 commented 3 years ago

That is a real head-scratcher. You have auth permission for cloud storage, and it looks like the same service email has access to Cloud Build but it gets a 403 error.

Do you have any builds in your history here? https://console.cloud.google.com/cloud-build/builds

I'm trying to eliminate if its permission to connect with Cloud Build API locally, or permission for the build to do things.

MarkEdmondson1234 commented 2 years ago

I think this was due to being an older GCP project, which doesn't come with the more modern service accounts etc. Will close but reopen if it comes up again.