Bucket resource fails

[turkenh]

What happened?

Trying to create a bucket resource with the provided example does not work. It waits for a while and eventually fails with:

Events:
  Type     Reason                         Age                    From                                                         Message
  ----     ------                         ----                   ----                                                         -------
  Warning  CannotObserveExternalResource  2m26s (x2 over 5m26s)  managed/storage.gcp.jet.crossplane.io/v1alpha1, kind=bucket  cannot run refresh: refresh failed:

With the same credentials and provider config, the instance example works. So, it is not a provider configuration issue.

Related slack discussion: https://crossplane.slack.com/archives/C01718T2476/p1638544968409900

How can we reproduce it?

Try to create an example bucket using manifest under examples dir.

What environment did it happen in?

Crossplane version: v1.5.0 Provider version: v0.1.0

[turkenh]

Looks like we are configuring bucket external name incorrectly: https://github.com/crossplane-contrib/provider-jet-gcp/blob/main/config/storage/config.go#L25

id field for bucket is set as bucket name only not project/bucket-name

[turkenh]

☝️ is one issue, however, it didn't fix the problem here.

Tried with terraform cli only and it hangs with:

bash-5.1$ terraform apply --refresh-only
google_storage_bucket.example: Refreshing state... [id=crossplane-example-bucket-hasan-test43]

I am wondering if this is something specific to refreshing a non-existing bucket 🤔

[turkenh]

By enabling debug logging I got more information. It looks like, different than other resources, with bucket resource terraform hangs (keeps retrying) with the following logs during our first refresh (i.e. when bucket does not exist) here:

@muvaf @ulucinar have you ever observed such a behavior?

{"error":{"code":404,"message":"The specified bucket does not exist.","errors":[{"message":"The specified bucket does not exist.","domain":"global","reason":"notFound"}]}}: timestamp=2021-12-07T12:01:40.655+0300
2021-12-07T12:01:40.655+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:40 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-12-07T12:01:40.655+0300
2021-12-07T12:01:40.655+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:40 [DEBUG] Dismissed an error as retryable. Retry 404s for bucket creation - googleapi: Error 404: The specified bucket does not exist., notFound: timestamp=2021-12-07T12:01:40.655+0300
2021-12-07T12:01:40.655+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:40 [TRACE] Waiting 10s before next try: timestamp=2021-12-07T12:01:40.655+0300
2021-12-07T12:01:50.656+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [INFO] Instantiating Google Storage client for path https://storage.googleapis.com/storage/v1/: timestamp=2021-12-07T12:01:50.656+0300
2021-12-07T12:01:50.657+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Retry Transport: starting RoundTrip retry loop: timestamp=2021-12-07T12:01:50.657+0300
2021-12-07T12:01:50.657+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Retry Transport: request attempt 0: timestamp=2021-12-07T12:01:50.657+0300
2021-12-07T12:01:50.657+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Google API Request Details:
---[ REQUEST ]---------------------------------------
GET /storage/v1/b/crossplane-example-bucket-hasan-test43?alt=json&prettyPrint=false HTTP/1.1
Host: storage.googleapis.com
User-Agent: google-api-go-client/0.5 Terraform/1.0.4 (+https://www.terraform.io) Terraform-Plugin-SDK/2.5.0 terraform-provider-google/4.2.1
X-Goog-Api-Client: gl-go/1.16.2 gdcl/20211027
Accept-Encoding: gzip

-----------------------------------------------------: timestamp=2021-12-07T12:01:50.657+0300
2021-12-07T12:01:50.925+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 404 Not Found
Content-Length: 171
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Tue, 07 Dec 2021 09:01:50 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: ADPycdvX7jeBOjawk2qHNJVjzZeci2qqgkL-sqzFggP_OWRvHSJ4QW1y4nbwAU534eoBWgKgDJoi10algE9eZSWCgQM

{
 "error": {
  "code": 404,
  "message": "The specified bucket does not exist.",
  "errors": [
   {
    "message": "The specified bucket does not exist.",
    "domain": "global",
    "reason": "notFound"
   }
  ]
 }
}
-----------------------------------------------------: timestamp=2021-12-07T12:01:50.925+0300
2021-12-07T12:01:50.925+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 404 with body: HTTP/2.0 404 Not Found
Content-Length: 171
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Type: application/json; charset=UTF-8
Date: Tue, 07 Dec 2021 09:01:50 GMT
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: UploadServer
Vary: Origin
Vary: X-Origin
X-Guploader-Uploadid: ADPycdvX7jeBOjawk2qHNJVjzZeci2qqgkL-sqzFggP_OWRvHSJ4QW1y4nbwAU534eoBWgKgDJoi10algE9eZSWCgQM

{"error":{"code":404,"message":"The specified bucket does not exist.","errors":[{"message":"The specified bucket does not exist.","domain":"global","reason":"notFound"}]}}: timestamp=2021-12-07T12:01:50.925+0300
2021-12-07T12:01:50.925+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-12-07T12:01:50.925+0300
2021-12-07T12:01:50.925+0300 [INFO]  provider.terraform-provider-google_v4.2.1_x5: 2021/12/07 12:01:50 [DEBUG] Dismissed an error as retryable. Retry 404s for bucket creation - googleapi: Error 404: The specified bucket does not exist., notFound: timestamp=2021-12-07T12:01:50.925+0300

[turkenh]

Hmm this looks related: https://github.com/hashicorp/terraform-provider-google/issues/10423

[muvaf]

Just wondering, what would it take to copy controller implementation of Bucket from provider-gcp? Like what changes would we possibly need to make that work?

[turkenh]

Just opened #16 as a workaround. I believe we could even make it better (with no schema change) with a terrajet change which could enable setting name parameter in main.tf from metadata.name without enabling nameAsExternal name initializer with something like:

-- type SetIdentifierArgumentsFn func(base map[string]interface{}, externalName string)
++ type SetIdentifierArgumentsFn func(base map[string]interface{}, tr resource.Terraformed)

I don't think we should make a change in core terrajet if there are no other cases that this could be helpful though.

[turkenh]

Just wondering, what would it take to copy controller implementation of Bucket from provider-gcp? Like what changes would we possibly need to make that work?

@muvaf with this resource being alpha, I believe we could live with the workaround until terraform fixes the issue.

However, we can also use this issue as an opportunity to experiment that as an option to tackle these kinds of issues in the future.

[turkenh]

Opened a fix on terraform (magic-modules/terraform) side: https://github.com/GoogleCloudPlatform/magic-modules/pull/5542

[turkenh]

Workaround for this issue is to add the following annotation as in the provided example.

terrajet.crossplane.io/provider-meta: '{"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0":{"create":60000000000,"read":60000000000}}'