google_dataflow_flex_template_job state becomes invalid after upgrading provider from 4.18 to 4.78

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.
If an issue is assigned to a user, that user is claiming responsibility for the issue.
Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v0.14.9 on Linux

provider registry.terraform.io/hashicorp/google-beta v4.78

Affected Resource(s)

google_dataflow_flex_template_job

Terraform Configuration

terraform {
  required_version = ">=0.14"
  required_providers {
    google-beta = "= 4.18"
  }
}

resource "google_dataflow_flex_template_job" "streaming_pipeline_synth_source" {
  provider                = google-beta
  project                 = "<your project>"
  name                    = "foo"
  container_spec_gcs_path = "gs://path-to-template.json"
  parameters = {
    experiments           = "upload_graph"
    usePublicIps          = false
    maxNumWorkers         = "2"
    enableStreamingEngine = true

    workerMachineType = "n1-highmem-4"
  }
  on_delete = "drain"
  region    = "us-central1"
}

Debug Output

Error: json: error calling MarshalJSON for type googleapi.RawMessage: unexpected end of JSON input

Expected Behavior

Terraform apply should still work.

Actual Behavior

The error "error calling MarshalJSON" is shown.

Steps to reproduce

terraform apply which succeeds
Change google-beta provider version to 4.78
terraform init -upgrade then terraform apply again which results in the MarshalJSON error.

Important Factoids

No response

References