Cannot run job automatically once it is created

antoinebourayne commented 1 year ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

Terraform v1.3.2

Affected Resource(s)

google_cloud_run_v2_job

Terraform Configuration Files

resource "google_cloud_run_v2_job" "test_job" {
  name         = "foo"
  project     = "project_id"
  location     = "some_region"
  launch_stage = "BETA"

  template {
    task_count  = 1
    parallelism = 1
    template {
      max_retries = 0
      timeout     = "10s"
      containers {
        image = "my_image"
      }
    }
  }
}

Expected Behavior

This resource should have the possibility to run the job on its creation.

Actual Behavior

It seems that you can't configure it to run once on creation.

It would be very useful to have a parameter to automatically run the job once it is created.

Equivalent in gcloud CLI would be the --execute-now flag.

References

b/272363295

edwardmedia commented 1 year ago

@antoinebourayne can you share the debug log for gcloud CLI execution?

danfundnel commented 1 year ago

@edwardmedia I can't speak for @antoinebourayne but I don't think there is actually any issue to debug here, the code posted works as intended but it will not trigger the job on creation. This is perhaps a feature request rather than a bug. A pretty key feature request though given that this is quite easy to do via the GCP CLI or GUI as mentioned.

edwardmedia commented 1 year ago

@antoinebourayne I think I understand what you want. But unfortunately it might not be a good fit in the model. Keep in mind, Terraform is a declarative type of tool while gcloud is a imperative. Does this make sense?

michaelpetri commented 1 year ago

I'm currently facing the same issue, and tried to use terraform-google-modules/terraform-google-gcloud in order to execute gcloud beta run jobs execute ${ local.name } --wait --region=${ var.google.region } but i'm always run into different errors.

Then i tried to use null_resource with local-exec but this didn't work either since the resource was not ready, at least i think so, because i accidentally used a hardcoded job name at some point and then all iterations over my module executed the same job and this worked. All over this is really hard to debug.

@antoinebourayne I think I understand what you want. But unfortunately it might not be a good fit in the model. Keep in mind, Terraform is a declarative type of tool while gcloud is a imperative. Does this make sense?

Not sure but from a declarative perspective i want that the job gets executed after it was created/updated? As an alternative, how about a google_cloud_run_v2_job_execution resource, would this work from your point of view?

rileykarson commented 1 year ago

As an alternative, how about a google_cloud_run_v2_job_execution resource, would this work from your point of view?

Doing this seems reasonable and would probably make more sense than adding this in-band on the resource, as "create this resource and trigger this action" can be a difficult lifecycle for Terraform to handle if the action execution fails. That often means tainting the resource, which can make it difficult to interact with.

michaelpetri commented 1 year ago

Just for the records how i've implemented this right now:

# Trigger execution, if requested
resource "null_resource" "job_execution_now" {
  count = var.application.job.execute_now ? 1 : 0

  provisioner "local-exec" {
    command = "gcloud beta run jobs execute ${ local.name } ${ var.application.job.wait ? "--wait" : ""  } --region=${ var.google.region }"
  }

  triggers = {
    always_run = timestamp()
  }

  depends_on = [google_cloud_run_v2_job.this]
}

With this i can execute pre-deployment jobs in a blocking and also execute my new deployments in async way.

gnarea commented 1 year ago

I tried to implement a generic module that would do this automatically when the job is created or updated, but I didn't manage to make the http resource make a POST request conditionally as I couldn't work around the error "The count value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created".

I'll abandon this approach, but in case anyone wants to have a go, here's what I have:

// Assumes you have a job named google_cloud_run_v2_job.bootstrap

resource "google_service_account" "bootstraper" {
  project = var.project_id

  account_id  = "awala-endpoint-${random_id.resource_suffix.hex}-boot"
  description = "Used to run bootstrapping job"
}

resource "google_cloud_run_v2_job_iam_member" "bootstrapper" {
  project  = google_cloud_run_v2_job.bootstrap.project
  location = google_cloud_run_v2_job.bootstrap.location
  name     = google_cloud_run_v2_job.bootstrap.name
  role     = "roles/run.invoker"
  member   = "serviceAccount:${google_service_account.bootstraper.email}"
}

resource "google_service_account_iam_binding" "bootstraper_impersonation" {
  service_account_id = google_service_account.bootstraper.id
  role               = "roles/iam.serviceAccountTokenCreator"
  members = [
    "serviceAccount:${data.google_client_openid_userinfo.me.email}",
  ]
}

resource "time_sleep" "wait_for_bootstraper_impersonation" {
  depends_on      = [google_service_account_iam_binding.bootstraper_impersonation]
  create_duration = "10s"
}

data "google_service_account_access_token" "bootstrapper" {
  target_service_account = google_service_account.bootstraper.email
  scopes                 = ["userinfo-email", "cloud-platform"]

  depends_on = [time_sleep.wait_for_bootstraper_impersonation]
}

resource "null_resource" "detect_bootstrap_job_change" {
  triggers = {
    job_generation = google_cloud_run_v2_job.bootstrap.generation
  }
}

data "http" "bootstrap_executer" {
  // Only execute when job changes. THIS IS WHAT FAILS.
  count = null_resource.detect_bootstrap_job_change.id == null ? 0 : 1

  url    = "https://${var.region}-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/${var.project_id}/jobs/${google_cloud_run_v2_job.bootstrap.name}:run"
  method = "POST"

  request_headers = {
    "Authorization" = "Bearer ${data.google_service_account_access_token.bootstrapper.access_token}"
    "Content-Type"  = "application/json"
  }

  retry {
    attempts     = 1
    min_delay_ms = 3000
  }

  lifecycle {
    postcondition {
      condition     = self.status_code == 200
      error_message = "Bootstrap execution was rejected"
    }
    postcondition {
      condition     = google_cloud_run_v2_job.bootstrap.generation == lookup(lookup(lookup(jsondecode(self.response_body), "metadata"), "labels"), "run.googleapis.com/jobGeneration")
      error_message = "Executed wrong generation of bootstrap job"
    }
  }
}

An alternative I considered but didn't have time to explore was to integrate Eventarc, so we listen for relevant Cloud Run events (e.g., CreateJob, UpdateJob), and then execute the job ASAP -- hopefully before the new service revisions start serving traffic.

whaught commented 1 month ago

Users should now be able to use the field start_execution_token or run_execution_token when declaring or updating the Job. The Job will create and run an execution (and either block on starting the execution, or running it to completion).

Updating the field to a different unique string will create a new run of the job.

rileykarson commented 1 month ago

Closing based on https://github.com/hashicorp/terraform-provider-google/issues/13390#issuecomment-2348911791

stebranchi commented 1 month ago

Hi everyone,

I just tried today to setup a cloud run job to be automatically executed when created using the attribute "start_execution_token", but my terraform apply failed telling me that the google cloud run job failed during his execution. Knowing that the cloud run job failure was caused by the code executed in the job and not terraform doing something wrong I wanted to know if this behaviour is intended or not?

Thank you in advance. Regards

github-actions[bot] commented 1 day ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-google