search

GoogleCloudPlatform / workflows-demos

A collection of Workflows samples for various use cases
Apache License 2.0
135 stars 37 forks source link

Create a sample for event-triggered Cloud Run Jobs leveraging execution override #99

Closed steren closed 1 year ago

steren commented 1 year ago

We need a YAML sample and instructions to set up a Workflow that executes a Cloud Run job, passing the events payload as environment variables via Execution overrides.

meteatamel commented 1 year ago

Hi @steren, there's already an Eventarc+Workflows+CloudRun jobs tutorial and sample.

Tutorial: https://cloud.google.com/workflows/docs/tutorials/execute-cloud-run-jobs

Sample: Workflows piece: https://github.com/GoogleCloudPlatform/workflows-demos/blob/master/cloud-run-jobs/workflow.yaml Cloud Run jobs piece: https://github.com/GoogleCloudPlatform/jobs-demos/blob/main/parallel-processing/process.py

Are you suggesting to update this tutorial/sample to use execution overrides or are you suggesting a new sample/tutorial?

Assuming it's the former, the sample currently has the bucket and file name hardcoded. Are you suggesting to update it so that the bucket name and file name are passed from event to Workflows to Cloud Run jobs as env variables execution overrides?

Note: I don't own these 2 samples, did/should we check with the sample owners first?

steren commented 1 year ago

yes, update this tutorial to pass the event attributes as overrides in the execution of the job, and make sure the jonb container is reading these values instead of using hardcoded balues

meteatamel commented 1 year ago

@steren, Ok, I'll look into this and update here once I have something.

meteatamel commented 1 year ago

@steren, mind approving the pull request on the Cloud Run jobs repo? https://github.com/GoogleCloudPlatform/jobs-demos/pull/39

meteatamel commented 1 year ago

Sample updated in both repos. I'll open a separate bug doc for the tutorial.