sasemb commented 3 years ago

Hi Team,

We are building google app engine both standard and flexible version using terraform .

It was working until yesterday . All of sudden when we ran the terraform code , during the flexible app deployment , Terraform thrown the below error:

Affected Resource(s)

google_app_engine_flexible_app_version

Terraform Block:

resource "google_app_engine_flexible_app_version" "myapp_v1" {
  count = "${var.apptype == "flex" ? 1 : 0}"
  version_id = "v1"
  service    = var.service
  runtime    = var.runtime
  #noop_on_destroy = false
  delete_service_on_destroy = true
  timeouts {
    create = "30m"
    delete = "30m"
    update = "30m"
  }
  entrypoint {

     shell =  var.entrypoint
  }

  deployment {
    zip {

      source_url  = "https://storage.googleapis.com/${google_storage_bucket_object.app-engine-source-zip-obj.bucket}/${google_storage_bucket_object.app-engine-source-zip-obj.name}"

    }
  }

   resources {
        cpu = var.cpu
        memory_gb = var.memory
        disk_gb = var.disk     
    }

  automatic_scaling {
    #min_idle_instances = 1
    min_total_instances = 1
    max_total_instances = 2
    cool_down_period = "120s"
    cpu_utilization {
      target_utilization = 0.5
    }
  }

  beta_settings = {
    cloud_sql_instances = var.cloud_sql_instance
  }

  liveness_check {
    path = var.livenesscheckpath

  }

  readiness_check {
    path = var.readinesscheckpath
    app_start_timeout = "4000s"
    success_threshold = 1

  }

}

Expected behaviour: Flexible app version should be created with new version

Actual Behavious: 2021-09-29T08:46:40.5847534Z [1m[31mError: [0m[0m[1mError updating FlexibleAppVersion "apps/qa-peso-na-measurement/services/nodejs/versions/v1": googleapi: Error 400: You have provided more than one input to an env: flex deployment (container, files). Please only select one source for your app code.[0m.

Steps to reproduce terraform apply Please note that we haven't done any changes recently to the above code . last change was couple of months back

edwardmedia commented 3 years ago

@sasemb below is the latest version that was released two day before you encountered the problem. From the release, I don't see any change that might cause the problem. From the error you provided, it could be caused by the changes either on your code or the api. Are you still able to repro the issue? If yes, could you provide the config (replace the var with real value) along with the debug log?

You have provided more than one input to an env: flex deployment (container, files). Please only select one source for your app code

https://github.com/hashicorp/terraform-provider-google/releases/tag/v3.86.0

sasemb commented 3 years ago

Hi ,

Yes the issue still persists and able to reproduce the error

Debug log

2021-10-08T12:48:13.7774835Z [0m[1mmodule.app-engine["nodejs"].google_storage_bucket_object.app-engine-source-zip-obj: Destroying... [id=qa-peso-na-measurement-us-src-app-engine-nodejs.3bc809abf143bd687cb180b5b5313ae0.zip][0m[0m 2021-10-08T12:48:13.7891994Z [0m[1mmodule.gcs_buckets.google_storage_bucket.buckets[1]: Modifying... [id=qa-peso-na-measurement-us-data][0m[0m 2021-10-08T12:48:13.7907957Z [0m[1mmodule.gcs_buckets.google_storage_bucket.buckets[0]: Modifying... [id=qa-peso-na-measurement-us-src][0m[0m 2021-10-08T12:48:14.0306563Z [0m[1mmodule.app-engine["nodejs"].google_storage_bucket_object.app-engine-source-zip-obj: Destruction complete after 0s[0m[0m 2021-10-08T12:48:14.0326616Z [0m[1mmodule.app-engine-default["default"].google_storage_bucket_object.app-engine-source-zip-obj: Destroying... [id=qa-peso-na-measurement-us-src-app-engine-default.8413b4cf4af4a6ce95ed5bfd6b1b242d.zip][0m[0m 2021-10-08T12:48:14.0942792Z [0m[1mmodule.gcs_buckets.google_storage_bucket.buckets[0]: Modifications complete after 0s [id=qa-peso-na-measurement-us-src][0m[0m 2021-10-08T12:48:14.1333809Z [0m[1mmodule.gcs_buckets.google_storage_bucket.buckets[1]: Modifications complete after 0s [id=qa-peso-na-measurement-us-data][0m[0m 2021-10-08T12:48:14.1461615Z [0m[1mmodule.app-engine-default["default"].data.archive_file.app-engine-source-zip: Reading... [id=87747d96cb3ea0eeec06edd6bad6203f63bd0bb9][0m[0m 2021-10-08T12:48:14.2315334Z [0m[1mmodule.app-engine-default["default"].google_storage_bucket_object.app-engine-source-zip-obj: Destruction complete after 0s[0m[0m 2021-10-08T12:48:14.4745893Z [0m[1mmodule.app-engine-default["default"].data.archive_file.app-engine-source-zip: Read complete after 0s [id=87747d96cb3ea0eeec06edd6bad6203f63bd0bb9][0m[0m 2021-10-08T12:48:14.4866324Z [0m[1mmodule.app-engine-default["default"].google_storage_bucket_object.app-engine-source-zip-obj: Creating...[0m[0m 2021-10-08T12:48:14.8461286Z [0m[1mmodule.app-engine-default["default"].google_storage_bucket_object.app-engine-source-zip-obj: Creation complete after 1s [id=qa-peso-na-measurement-us-src-app-engine-default.8413b4cf4af4a6ce95ed5bfd6b1b242d.zip][0m[0m 2021-10-08T12:48:14.8643881Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1][0m[0m 2021-10-08T12:48:24.8591966Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Still modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1, 10s elapsed][0m[0m 2021-10-08T12:48:34.8594687Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Still modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1, 20s elapsed][0m[0m 2021-10-08T12:48:44.8598123Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Still modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1, 30s elapsed][0m[0m 2021-10-08T12:48:54.8607912Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Still modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1, 40s elapsed][0m[0m 2021-10-08T12:49:04.8621521Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Still modifying... [id=apps/qa-peso-na-measurement/services/default/versions/v1, 50s elapsed][0m[0m 2021-10-08T12:49:05.7754799Z [0m[1mmodule.app-engine-default["default"].google_app_engine_standard_app_version.app-version[0]: Modifications complete after 51s [id=apps/qa-peso-na-measurement/services/default/versions/v1][0m[0m 2021-10-08T12:49:05.7776101Z [0m[1mmodule.app-engine["nodejs"].data.archive_file.app-engine-source-zip: Reading... [id=d271f622e4b2d99f4b1a67e2ddc77fbce6202c88][0m[0m 2021-10-08T12:49:15.7790514Z [0m[1mmodule.app-engine["nodejs"].data.archive_file.app-engine-source-zip: Still reading... [id=d271f622e4b2d99f4b1a67e2ddc77fbce6202c88, 10s elapsed][0m[0m 2021-10-08T12:49:15.8602577Z [0m[1mmodule.app-engine["nodejs"].data.archive_file.app-engine-source-zip: Read complete after 10s [id=dd000d0c37846206c5b55b86a0a9b9d6a1fbe41c][0m[0m 2021-10-08T12:49:15.9963719Z [0m[1mmodule.app-engine["nodejs"].google_storage_bucket_object.app-engine-source-zip-obj: Creating...[0m[0m 2021-10-08T12:49:17.3598928Z [0m[1mmodule.app-engine["nodejs"].google_storage_bucket_object.app-engine-source-zip-obj: Creation complete after 1s [id=qa-peso-na-measurement-us-src-app-engine-nodejs.94a831f1d3ef561048ae847d7bc8bf8d.zip][0m[0m 2021-10-08T12:49:17.3759108Z [0m[1mmodule.app-engine["nodejs"].google_app_engine_flexible_app_version.myapp_v1[0]: Modifying... [id=apps/qa-peso-na-measurement/services/nodejs/versions/v1][0m[0m 2021-10-08T12:49:18.0532069Z [33m 2021-10-08T12:49:18.0549256Z [1m[33mWarning: [0m[0m[1mInterpolation-only expressions are deprecated[0m 2021-10-08T12:49:18.0549923Z 2021-10-08T12:49:18.0550619Z [0m on modules\app-engine\main.tf line 17, in resource "google_app_engine_standard_app_version" "app-version": 2021-10-08T12:49:18.0551246Z 17: count = [4m"${var.apptype == "standard" ? 1 : 0}"[0m 2021-10-08T12:49:18.0551637Z [0m 2021-10-08T12:49:18.0552034Z Terraform 0.11 and earlier required all non-constant expressions to be 2021-10-08T12:49:18.0552557Z provided via interpolation syntax, but this pattern is now deprecated. To 2021-10-08T12:49:18.0553061Z silence this warning, remove the "${ sequence from the start and the }" 2021-10-08T12:49:18.0553574Z sequence from the end of this expression, leaving just the inner expression. 2021-10-08T12:49:18.0553954Z 2021-10-08T12:49:18.0554355Z Template interpolation syntax is still used to construct strings from 2021-10-08T12:49:18.0554866Z expressions when the template includes multiple interpolation sequences or a 2021-10-08T12:49:18.0555382Z mixture of literal strings and interpolations. This deprecation applies only 2021-10-08T12:49:18.0555889Z to templates that consist entirely of a single interpolation sequence. 2021-10-08T12:49:18.0556396Z 2021-10-08T12:49:18.0556896Z (and 3 more similar warnings elsewhere) 2021-10-08T12:49:18.0557395Z [0m[0m 2021-10-08T12:49:18.0568816Z [31m 2021-10-08T12:49:18.0569673Z [1m[31mError: [0m[0m[1mError updating FlexibleAppVersion "apps/qa-peso-na-measurement/services/nodejs/versions/v1": googleapi: Error 400: You have provided more than one input to an env: flex deployment (container, files). Please only select one source for your app code.[0m 2021-10-08T12:49:18.0571035Z 2021-10-08T12:49:18.0571729Z [0m on modules\app-engine\main.tf line 68, in resource "google_app_engine_flexible_app_version" "myapp_v1": 2021-10-08T12:49:18.0572363Z 68: resource "google_app_engine_flexible_app_version" "myapp_v1" [4m{[0m 2021-10-08T12:49:18.0572772Z [0m 2021-10-08T12:49:18.0573037Z [0m[0m 2021-10-08T12:49:18.2860650Z ##[error]Error: The process 'C:\hostedtoolcache\windows\terraform\0.14.0\x64\terraform.exe' failed with exit code 1 2021-10-08T12:49:18.2877748Z ##[section]Finishing: Terraform apply : gcp

########################################## Terraform code with variable replaced

resource "google_app_engine_flexible_app_version" "myapp_v1" { version_id = "v1" service = nodejsservice runtime = nodejs

noop_on_destroy = false

delete_service_on_destroy = true timeouts { create = "30m" delete = "30m" update = "30m" } entrypoint {

shell = "ls" #not required }

deployment { zip {

source_url = "https://storage.googleapis.com/qa-peso-na-measurement-us-src/app-engine-default.8413b4cf4af4a6ce95ed5bfd6b1b242d.zip"

} }

resources { cpu = 4 memory_gb = 4 disk_gb = 40 }

automatic_scaling {

min_idle_instances = 1

min_total_instances = 1 max_total_instances = 2 cool_down_period = "120s" cpu_utilization { target_utilization = 0.5 } }

beta_settings = { cloud_sql_instances = "peso-na-measurement-project:us-central1:peso-na-measurement" }

liveness_check { path = "/health_check"

}

readiness_check { path = "/health_check" app_start_timeout = "4000s" success_threshold = 1

}

sasemb commented 3 years ago

@edwardmedia did you get chance to see the above logs and configuration

edwardmedia commented 3 years ago

@sasemb are you able to set the environment variable export TF_LOG=DEBUG and then obtain the log?

Can you verify if the zip is available and accessible?

"https://storage.googleapis.com/qa-peso-na-measurement-us-src/app-engine-default.8413b4cf4af4a6ce95ed5bfd6b1b242d.zip"

peterfraedrich commented 3 years ago

I can confirm that I've been seeing this too on the latest versions of both Terraform and the provider.

edwardmedia commented 3 years ago

@sasemb @peterfraedrich Below is the error from what you provided. It does not tell us much what went wrong. Could you share the full debug log that contains all the api requests and responses as I suggested here?

updating FlexibleAppVersion "apps/qa-peso-na-measurement/services/nodejs/versions/v1": googleapi: Error 400: You have provided more than one input to an env: flex deployment (container, files). Please only select one source for your app code

sasemb commented 3 years ago

@edwardmedia I'm just waiting for the same error . Since it's sporadic issue , It didn't occur now but definitely it should happen in our upcoming deployments. I will share the logs as soon as I get the error in coming weeks. Please keep the issue open

edwardmedia commented 3 years ago

Waiting for the logs

sasemb commented 3 years ago

@edwardmedia Continously monitoring the deployments . Unfortunately some of the recent releases are passing and it's sporadic in nature . Will post the debug logs once we get the failure again

sasemb commented 3 years ago

12_Read the Debug Log.log So finally we got the error again and here I attached the complete DEBUG log . please let me know

sasemb commented 3 years ago

@edwardmedia added the logs . please check and let me know if you need any further information

edwardmedia commented 3 years ago

@sasemb I am seeing below two sources in the call. Did you say no config change on your side? Working, and suddenly not working? I wonder API behave a little weird. Provider sends sourceUrl in the request and API returns the reference to the image? Did you ever provide image?

2021-11-08T07:31:52.1314856Z  },
2021-11-08T07:31:52.1315325Z  "deployment": {
2021-11-08T07:31:52.1315839Z   "container": {
2021-11-08T07:31:52.1316544Z    "image": "us.gcr.io/qa-peso-na-measurement/appengine/nodejs.v1@sha256:c501f5ba7e3c4ec3d15d5e2655285e2fd4fa88640ab23c012032d0637e60a72a"
2021-11-08T07:31:52.1317246Z   },
2021-11-08T07:31:52.1317661Z   "zip": {
2021-11-08T07:31:52.1318372Z    "sourceUrl": "https://storage.googleapis.com/qa-peso-na-measurement-us-src/app-engine-nodejs.4f6a4b68aafccaa2a96325aac144be2e.zip"
2021-11-08T07:31:52.1319046Z   }
2021-11-08T07:31:52.1319471Z  },

sasemb commented 3 years ago

No , I haven't passed any image reference to this service and there was no change in terraform code from long time . It has only the sourceURL block . Adding the source code skeleton again for your reference below

Terraform code

resource "google_app_engine_flexible_app_version" "myapp_v1" { count = "${var.apptype == "flex" ? 1 : 0}" version_id = "v1" service = var.service runtime = var.runtime

noop_on_destroy = false

delete_service_on_destroy = true timeouts { create = "30m" delete = "30m" update = "30m" } entrypoint {

 shell =  var.entrypoint

}

deployment { zip {

  source_url  = "https://storage.googleapis.com/${google_storage_bucket_object.app-engine-source-zip- obj.bucket}/${google_storage_bucket_object.app-engine-source-zip-obj.name}"

}

}

resources { cpu = var.cpu memory_gb = var.memory disk_gb = var.disk
}

automatic_scaling {

min_idle_instances = 1

min_total_instances = 1
max_total_instances = 2
cool_down_period = "120s"
cpu_utilization {
  target_utilization = 0.5
}

}

beta_settings = { cloud_sql_instances = var.cloud_sql_instance }

liveness_check { path = var.livenesscheckpath

}

readiness_check { path = var.readinesscheckpath app_start_timeout = "4000s" success_threshold = 1

}

sasemb commented 3 years ago

@edwardmedia Added the code block above for your reference

edwardmedia commented 3 years ago

@peterfraedrich if image is not included in the deployment block, I am not sure where it comes from. I have tested the deployment by updating different versions of object (zip files), and can't repro your issue. Can you see if there are other code that may cause this? Can you create a separate config code that can repro the issue? We need step by step to verify and understand where the image comes from. The error message is clear. Does this make sense?

peterfraedrich commented 3 years ago

@edwardmedia from what I can tell it seems that when you upload a zip file Cloud Build takes that zip and puts it into an image, then substitutes the image URL for the ZIP file in the original App Engine config. This is what creates the mismatch issue where an image gets populated that Terraform didn't know about.

edwardmedia commented 3 years ago

@sasemb I see where the problem is. There was below PR released about month ago. I can repro the issue now with the latest version

https://github.com/hashicorp/terraform-provider-google/pull/10058

Here is the original issue https://github.com/hashicorp/terraform-provider-google/issues/9851

sasemb commented 3 years ago

@edwardmedia Can you provide the solution / recommendation . Do we need some version upgrades? Sorry I didnt get the resolution

sasemb commented 3 years ago

@edwardmedia Appreciate if you can help with the solution . Do we need to rollback to some previous versions?

c2thorn commented 2 years ago

@sasemb I apologize for the delay. To immediately work around this issue, I suggest pinning your provider version to v3.84.0 which is the release just before this change.

From what I can tell, the fix for this requires the provider to not send the container block if the files block exists. I'll get started on that.

shaakaud commented 2 years ago

Hi, I am also facing the same issue. Has this been fixed in the latest version v4.1.0 ?

sasemb commented 2 years ago

@c2thorn Can you please provide me the tentative date when this can be fixed?

Also for time being , I set the provider version like below . Please confirm if it's correct

provider "google" { project = var.project_id[var.env] region = var.location version = "3.84.0"

}

edwardsPaul421 commented 2 years ago

@sasemb I apologize for the delay. To immediately work around this issue, I suggest pinning your provider version to v3.84.0 which is the release just before this change.

From what I can tell, the fix for this requires the provider to not send the container block if the files block exists. I'll get started on that.

Hi, Im dealing with this same issue using v4.19. @c2thorn Is someone is working on fixing it?

edwardsPaul421 commented 2 years ago

@c2thorn @sasemb Is anybody working on fixing this issue?

hur commented 1 year ago

I am also having this issue, hoping for a fix so I don't have to rely on v3.84.0

sdiepend commented 1 year ago

Also still running into this issue it seems.

hashicorp / terraform-provider-google

##critical -- google_app_engine_flexible_app_version failing with 400 response #10185

Affected Resource(s)

noop_on_destroy = false

min_idle_instances = 1

noop_on_destroy = false

min_idle_instances = 1