"Remote state GCS bucket x does not exist. Attempting to create it" even though bucket does exist

thnee commented 5 years ago

I think this is an issue that started occurring after upgrading to 0.19 and terraform 0.12.

Problem

Terragrunt says that the state bucket does not exist, even though it does.
After that, there is an empty log line? Is there some error that happened that is not shown to me?
Then it says Missing required GCS remote state configuration project which I am guessing is just a side effect of the above point.

λ export TERRAGRUNT_SOURCE=/home/thnee/code/gitlab/lookatech/titanium/modules//bastion
λ export TERRAGRUNT_DEBUG=1
λ terragrunt apply         
[terragrunt] [/home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion] 2019/08/07 13:34:56 Running command: terraform --version
[terragrunt] 2019/08/07 13:34:56 Reading Terragrunt config file at /home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion/terragrunt.hcl
[terragrunt] 2019/08/07 13:34:56 Downloading Terraform configurations from file:///home/thnee/code/gitlab/lookatech/titanium/modules into /home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion/.terragrunt-cache/9a8XoLJGysH3UXy51GAkDY40dZs/6nM5s3aDYC3CplvXfsAEmgHOkTQ
[terragrunt] 2019/08/07 13:34:56 Copying files from /home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion into /home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion/.terragrunt-cache/9a8XoLJGysH3UXy51GAkDY40dZs/6nM5s3aDYC3CplvXfsAEmgHOkTQ/bastion
[terragrunt] 2019/08/07 13:34:56 Setting working directory to /home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion/.terragrunt-cache/9a8XoLJGysH3UXy51GAkDY40dZs/6nM5s3aDYC3CplvXfsAEmgHOkTQ/bastion
[terragrunt] [/home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion] 2019/08/07 13:34:56 Initializing remote state for the gcs backend
[terragrunt] [/home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion] 2019/08/07 13:34:56 Remote state GCS bucket alterra-cyclops-prod-terraform-state does not exist. Attempting to create it
[terragrunt] [/home/thnee/code/gitlab/lookatech/titanium/projects/cyclops-prod/bastion] 2019/08/07 13:34:56 
[terragrunt] 2019/08/07 13:34:56 remote.MissingRequiredGCSRemoteStateConfig Missing required GCS remote state configuration project
/go/src/github.com/gruntwork-io/terragrunt/remote/remote_state_gcs.go:218 (0xbbed48)
/go/src/github.com/gruntwork-io/terragrunt/remote/remote_state_gcs.go:141 (0xbbe334)
<autogenerated>:1 (0xbc6196)
/go/src/github.com/gruntwork-io/terragrunt/remote/remote_state.go:60 (0xbbcf12)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:482 (0xcd5e03)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:397 (0xcd55d3)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:603 (0xcd684a)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:546 (0xcd652b)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:401 (0xcd54b1)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:298 (0xcd433f)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:231 (0xcd3f01)
/go/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:212 (0xcd3bf5)
/go/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:490 (0xc714f8)
/go/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:264 (0xc6f76d)
/go/src/github.com/gruntwork-io/terragrunt/main.go:20 (0xcdbf94)
/usr/local/go/src/runtime/proc.go:201 (0x42c1b7)
/usr/local/go/src/runtime/asm_amd64.s:1333 (0x457fe1)

[terragrunt] 2019/08/07 13:34:56 Unable to determine underlying exit code, so Terragrunt will exit with error code 1

As you can see this bucket does exist and is accessible for me.

λ gsutil ls gs://alterra-cyclops-prod-terraform-state
λ echo $?
0

Sample code

prod/terragrunt.hcl

remote_state {
  backend = "gcs"

  config = {
    bucket = "alterra-cyclops-prod-terraform-state"
    prefix = path_relative_to_include()
  }
}

prod/bastion/terragrunt.hcl

include {
  path = find_in_parent_folders()
}

terraform {
  source = "git::ssh://git@gitlab.com/lookatech/titanium.git//modules/bastion"
}

inputs = {
    ...
}

robmorgan commented 5 years ago

Hoping to look tomorrow 👍

thnee commented 5 years ago

Thanks rob! Yeah I noticed you were the one who implemented #757, I was trying to read the code for DoesGCSBucketExist but not sure about what kind of details are going on there. I can't really figure out what else I should test here.. Anyway here are my versions:

λ terragrunt --version
terragrunt version v0.19.14
λ terraform --version
Terraform v0.12.6

thnee commented 5 years ago

I would really appreciate if there is some way to just completely disable the automatic bucket creation and existence check. Because I definitely intend to create the bucket myself anyway, and this issue is currently a blocker for me. Is there some way to do that?

robmorgan commented 5 years ago

@thnee there's actually another PR here: https://github.com/gruntwork-io/terragrunt/pull/818 I will also review tomorrow that allows the automatic bucket creation to be disabled.

thnee commented 5 years ago

I was able to temporarily work around this issue for now by downgrading to 0.19.5. Fortunately it seems I don't have any immediate need for anything added since then.

But I also want to note that I am now extra confused by this issue. Because 0.19.14 actually works fine in our demo environment. This issue only happens in our prod environment. There is basically no difference in the code between our demo and prod (except for the name of the state bucket, and the module inputs).

And both state buckets in demo and prod have identical settings and permissions, except for the name. I have verified this by comparing them in GCS, and also by the fact that they are created using the same terraform code. I am also able to access the bucket using gsutil, and all my prod code works when I downgrade to terragrunt 0.19.5, so it does not seem like there is a permission or spelling mistake in my code or anything like that.

The main difference is that prod was never actually provisioned before, meaning that there were no resources created, and the state bucket was empty. Perhaps that is a cause for this issue?

Sorry for being long winded, just trying to provide what info I can.

thnee commented 5 years ago

Yep, I downgraded to 0.19.5, provisioned the module once, and then graded back up 0.19.14, and then it works fine. So it definitely seems like an empty bucket is not being handled correctly.

thnee commented 5 years ago

No, actually, strike that. I tried deleting everything in the bucket, and 0.9.14 still works fine.

I think the problem was that the API access was not enabled.

When I downgraded to 0.19.5 I got this message.

Error: Failed to get existing workspaces: querying Cloud Storage failed: googleapi: Error 403: Access Not Configured. Cloud Storage JSON API has not been used in project XXX before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/storage-api.googleapis.com/overview?project=XXX then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

This message was never shown when using 0.19.14, but now that I have enabled the API access, 0.19.14 works.

robmorgan commented 5 years ago

Hi @thnee could you upgrade to https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.19 and let me know if this works for you? This section in the README explains how to disable the bucket creation + check.

thnee commented 5 years ago

I'm sure the disable mechanism in 0.19.19 works fine, but I don't really need it now that I have worked around the problem in a different way, by enabling the Cloud Storage JSON API.

But I would still say that this issue represents a bug.

Previously (i.e. in 0.19.5), terragrunt would display a nice and helpful error message when the Cloud Storage JSON API was not enabled, including a helpful link where the user can enable it. In recent versions, it doesn't.

Instead, it fails with a message that is not true (bucket x does not exist). This should at least be changed to something like terragrunt was unable to determine if the bucket exists. But preferably, the old message should be brought back, cause is very helpful.

sc250024 commented 5 years ago

UPDATE Adding skip_bucket_creation = true to the parent terragrunt.hcl file seemed to do the trick!

Same issue here. Here's my information:

Terragrunt shell debug output

```shell $ TERRAGRUNT_DEBUG=1 terragrunt apply [terragrunt] 2019/08/12 15:20:01 Reading Terragrunt config file at /path/to/my/terraform-gcp/projects/sandbox/terragrunt.hcl [terragrunt] [/path/to/my/terraform-gcp/projects/sandbox] 2019/08/12 15:20:01 Running command: terraform --version [terragrunt] 2019/08/12 15:20:02 Skipping var-file ignore as it does not exist [terragrunt] [/path/to/my/terraform-gcp/projects/sandbox] 2019/08/12 15:20:02 Initializing remote state for the gcs backend [terragrunt] [/path/to/my/terraform-gcp/projects/sandbox] 2019/08/12 15:20:02 Remote state GCS bucket my-terraform-state-bucket does not exist. Attempting to create it [terragrunt] [/path/to/my/terraform-gcp/projects/sandbox] 2019/08/12 15:20:02 my-gcp-project [terragrunt] 2019/08/12 15:20:02 remote.MissingRequiredGCSRemoteStateConfig Missing required GCS remote state configuration location /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/errors/errors.go:15 (0x1864701) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/remote/remote_state_gcs.go:146 (0x1863dda) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/remote/remote_state.go:60 (0x18628bf) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:495 (0x1982260) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:410 (0x19819d4) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:616 (0x1982c88) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:559 (0x1982958) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:414 (0x19818b2) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:311 (0x19807cf) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:225 (0x19802a3) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/cli/cli_app.go:206 (0x197ffe5) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:490 (0x191a6d8) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/vendor/github.com/urfave/cli/app.go:264 (0x19188f0) /private/tmp/terragrunt-20190808-90179-7gg9jp/src/github.com/gruntwork-io/terragrunt/main.go:20 (0x1988b54) /usr/local/Cellar/go/1.12.7/libexec/src/runtime/proc.go:200 (0x102f67c) main: fn() /usr/local/Cellar/go/1.12.7/libexec/src/runtime/asm_amd64.s:1337 (0x105ace1) goexit: BYTE $0x90 // NOP [terragrunt] 2019/08/12 15:20:02 Unable to determine underlying exit code, so Terragrunt will exit with error code 1 $ gsutil ls gs://my-terraform-state-bucket/ ```

terragrunt.hcl (parent)

```hcl remote_state { backend = "gcs" config = { bucket = "my-terraform-state-bucket" credentials = "${get_env("TERRAFORM_GOOGLE_APPLICATION_CREDENTIALS", "${get_parent_terragrunt_dir()}/credentials.json")}" prefix = "${path_relative_to_include()}" project = "my-gcp-project" } } terraform { extra_arguments "conditional_vars" { commands = get_terraform_commands_that_need_vars() required_var_files = [ "${get_parent_terragrunt_dir()}/global.tfvars" ] optional_var_files = [ "${find_in_parent_folders("environment.tfvars", "ignore")}", "${find_in_parent_folders("account.tfvars", "ignore")}", "${find_in_parent_folders("region.tfvars", "ignore")}", "${find_in_parent_folders("stage.tfvars", "ignore")}" ] } extra_arguments "disable_input" { commands = get_terraform_commands_that_need_input() arguments = ["-input=false"] } extra_arguments "retry_lock" { commands = get_terraform_commands_that_need_locking() arguments = ["-lock-timeout=5m"] } } ```

terragrunt.hcl (child)

```hcl include { path = find_in_parent_folders() } ```

Terrform version: Terraform v0.12.6 Terragrunt version: terragrunt version v0.19.19

gruntwork-io / terragrunt