Closed MarkEdmondson1234 closed 2 years ago
This use case covers a Cloud Run version, but Cloud Build would be better for longer batch jobs https://code.markedmondson.me/googleCloudRunner/articles/usecase-r-event-driven-pubsub.html
Cloud Build triggering from a Cloud Scheduler job would be better than using the current HTTP API call functionality.
Cloud Scheduler -> Pub/Sub topic -> Pub/Sub subscription -> Cloud Build
Could use dependency on googlePubsubR
to auto-create the topics/subs
https://github.com/andodet/googlePubsubR
As well as scheduling, can trigger via normal pubsub and use data in the message to add variables to the build.
# create build object
cloudbuild <- system.file("cloudbuild/cloudbuild_substitutions.yml",
package = "googleCloudRunner")
the_build <- cr_build_make(cloudbuild)
# create pub/sub topic if needed
library(googlePubsubR)
pubsub_auth()
topics_create("test-topic")
# create build trigger that will work from pub/subscription
pubsub_trigger <- cr_buildtrigger_pubsub("test-topic")
cr_buildtrigger(the_build, name = "pubsub-triggered-subs", trigger = pubsub_trigger)
# make base64 encoded json
library(jsonlite)
message <- list(var1 = "hello mum")
send_me <- base64_enc(toJSON(message))
topics_publish(PubsubMessage(send_me), "test-topic")
# did it work?
cr_build_logs_last("pubsub-triggered-subs")
Yep!
[51] "Step #1 - \"Hello R\": 111255bd9d45: Pull complete"
[52] "Step #1 - \"Hello R\": 06ab1c1acd68: Pull complete"
[53] "Step #1 - \"Hello R\": 943eb4dce0a5: Pull complete"
[54] "Step #1 - \"Hello R\": Digest: sha256:9cb64b052b519bdeee81f1bda9606dad826c5d4dd135a3acadf3fad23b05e5fe"
[55] "Step #1 - \"Hello R\": Status: Downloaded newer image for rocker/r-base:latest"
[56] "Step #1 - \"Hello R\": docker.io/rocker/r-base:latest"
[57] "Step #1 - \"Hello R\": [1] \"From PubSub message field1:[\\\"hello mum\\\"]\""
[58] "Finished Step #1 - \"Hello R\""
[59] "PUSH"
[60] "DONE"
[61] ""
Test set up
cloudbuild <- system.file("cloudbuild/cloudbuild_substitutions.yml",
package = "googleCloudRunner")
the_build <- cr_build_make(cloudbuild)
# var1 is sent via Pubsub to the buildtrigger
message <- list(var1 = "hello mum")
send_me <- googlePubsubR::msg_encode(jsonlite::toJSON(message))
# create build trigger that will work from pub/subscription
pubsub_trigger <- cr_buildtrigger_pubsub("test-topic", projectId = "learning-ga4")
cr_buildtrigger(the_build, name = "pubsub-triggered-subs-param", trigger = pubsub_trigger, projectId = "learning-ga4", overwrite = TRUE)
# create scheduler that calls the pub/sub topic with a parameter
cr_schedule("cloud-build-pubsub-params",
"15 5 * * *",
pubsubTarget = cr_schedule_pubsub("test-topic",
data = send_me, projectId = "learning-ga4"), projectId = "learning-ga4", overwrite = TRUE)
Do we have examples and support for this? A pub/sub trigger for when a docker container is built and/or Cloud Storage files are ready is a good utility.
https://cloud.google.com/build/docs/automating-builds/create-pubsub-triggers