MarkEdmondson1234 / googleCloudRunner

Easy R scripts on Google Cloud Platform via Cloud Run, Cloud Build and Cloud Scheduler
https://code.markedmondson.me/googleCloudRunner/
Other
81 stars 26 forks source link

Move Cloud Scheduler builds to use PubSub not HTTP #148

Open MarkEdmondson1234 opened 2 years ago

MarkEdmondson1234 commented 2 years ago

More reliable, better utility since you can have other subscriptions attached to the topic the scheduler hits, and less mucking around with authentication for the Cloud Scheduler (big plus)

https://github.com/MarkEdmondson1234/googleCloudRunner/issues/140

MarkEdmondson1234 commented 2 years ago

Workflow will need to create the Cloud Build first, rather than having it in the API call directly so still a use case for the former I guess. But a deploy function would do following:

# create build object
cloudbuild <- system.file("cloudbuild/cloudbuild.yaml",
                           package = "googleCloudRunner")
the_build <- cr_build_make(cloudbuild)

# create pub/sub topic
library(googlePubsubR)
pubsub_auth()
topics_create("test-topic")

# create build trigger that will work from pub/subscription
pubsub_trigger <- cr_buildtrigger_pubsub("test-topic")
cr_buildtrigger(the_build, name = "pubsub-triggered", trigger = pubsub_trigger)

# create scheduler that calls the pub/sub topic
cr_schedule("cloud-build-pubsub", "15 5 * * *",
                      pubsubTarget = cr_build_schedule_pubsub("test-topic"))
MarkEdmondson1234 commented 2 years ago

A new role needed?

Error: API returned: Caller does not have required permission to use project xxx. 
Grant the caller the roles/serviceusage.serviceUsageConsumer role, 
or a custom role with the serviceusage.services.use permission,....
MarkEdmondson1234 commented 2 years ago

The gcloud fails too, maybe old project needing a new service account?

gcloud alpha builds triggers create pubsub --name=pubsub-gcloud-trigger --topic=projects/xxx/topics/test-topic --build-config=inst/cloudbuild/cloudbuild.yaml --repo=MarkEdmondson1234/googleCloudRunner --branch=master
ERROR: (gcloud.alpha.builds.triggers.create.pubsub) PERMISSION_DENIED: Caller does not have required permission to use project xxx. Grant the caller the roles/serviceusage.serviceUsageConsumer role, or a custom role with the serviceusage.services.use permission, by visiting https://console.developers.google.com/iam-admin/iam/project?project=xxx and then retry. Propagation of the new permission may take a few minutes.
- '@type': type.googleapis.com/google.rpc.Help
  links:
  - description: Google developer console IAM admin
    url: https://console.developers.google.com/iam-admin/iam/project?project=xxx
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: googleapis.com
  metadata:
    consumer: projects/xxx
    service: pubsub.googleapis.com
  reason: USER_PROJECT_DENIED

And in the UI, so I guess something project specific - needs anotehr Google service email added or something

image
MarkEdmondson1234 commented 2 years ago

It all works ok now in new GCP project

MarkEdmondson1234 commented 2 years ago

Turning the Cloud Build API off and on again on old projects looks to fix not being able to use PubSub triggers issue.

MarkEdmondson1234 commented 2 years ago

For PubSub triggers I need to add the new property sourceToBuild otherwise there is nothing to build :)

https://cloud.google.com/build/docs/api/reference/rest/v1/projects.locations.triggers#gitreposource