Error: timed out while waiting for user to be inserted

[xens]

Hi there,

I'm hitting a random but very frequent issue when creating Google Workspace users. The user creation process takes usually a lot of time (5 minutes per user) and most of the time it times out.

Terraform Version

$ terraform -v
Terraform v1.3.4
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v4.42.0
+ provider registry.terraform.io/hashicorp/googleworkspace v0.7.0
+ provider registry.terraform.io/hashicorp/random v3.4.3
+ provider registry.terraform.io/hashicorp/time v0.9.0

Affected Resource(s)

• googleworkspace_user

Terraform Configuration Files

resource "random_password" "password" {
  length = 30
  special = true
}

resource "googleworkspace_user" "users" {

  primary_email = "test@testdomain.com"
  password      = md5(random_password.password.result)
  hash_function = "MD5"

  org_unit_path = "/myOrg"

  name {
    family_name = "userFamiliyName"
    given_name  = "userGivenName"
  }

  organizations {
    department = "my Dept"
    primary    = true
    type       = "work"
  }
  recovery_email = "test@testdomain2.com"

    timeouts {
    create = "15m"
  }

Expected Behavior

User should be properly created.

Actual Behavior

90% of the time the user creation fails with the following error

googleworkspace_user.users["test@testdomain.com"]: Still creating... [6m20s elapsed]
googleworkspace_user.users["test@testdomain.com"]: Still creating... [6m30s elapsed]
googleworkspace_user.users["test@testdomain.com"]: Still creating... [6m40s elapsed]
googleworkspace_user.users["test@testdomain.com"]: Still creating... [6m50s elapsed]
│ Error: timed out while waiting for user to be inserted
│ 
│   with googleworkspace_user.users["test@testdomain.com"],
│   on users.tf line 14, in resource "googleworkspace_user" "users":
│   14: resource "googleworkspace_user" "users" {

Steps to Reproduce

1. terraform apply

[xens]

I'll need to sanitize a bit my TF_LOG before uploading it but I've been able to dig a bit more into this issue already and looks like its linked to the resource consistencyCheck not being met.

According to this file, the number of consistent responses from the Google Workspace API must equal 4.

// The number of consistent responses we want before we consider the resource consistent
const numConsistent = 4

And it uses the response etag value to make these consistency checks.

By grep'ing the responses etags in TF_LOG=trace we can see that its value is often flipping between two different values and thus not meeting the consistencyCheck condition.

examples when it errors:

 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/-uCe4x3GsEUVT6jHRPQf0qnDKKE\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/v9L9yvP2KDdjQO8ORY-9L3wlytw\"",

And when it succeeds we can see that the condition is met

 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/q6_2E4VfX6skeuoFnf4_SITEGww\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"ncll-7bPS7lrDES-QUXBlfs2Pot1Y168LPxnrGE6FJU/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/WMrMd3cEhdhF_9eF63F5_pItbJU\"",

[xens]

Wondering if it could be linked to https://github.com/hashicorp/terraform-provider-googleworkspace/issues/263

[xens]

I did some more tests today and it looks like that Google APIs are returning more consistent etags this morning and thus the creation process succeeds more frequently

googleworkspace_user.users["test1@testdomain.com"]: Creation complete after 1m48s [id=103959683524035821588]
googleworkspace_user.users["test2@testdomain.com"]: Creation complete after 1m48s [id=114270762350499200959]
googleworkspace_user.users["test3@testdomain.com"]: Creation complete after 1m48s [id=106563825273091153419]
googleworkspace_user.users["test4@testdomain.com"]: Creation complete after 1m58s [id=101969177768374119408]

Apply complete! Resources: 8 added, 0 changed, 1 destroyed.

Outputs:

user_email = [
  "test1@testdomain.com",
  "test2@testdomain.com",
  "test3@testdomain.com",
  "test4@testdomain.com",
]

and of course I can still correlate the creation successes with a series of consistent etags corresponding to the resource.

grep "\"etag" out.log | grep "qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU" 
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",
 "etag": "\"qUvL9RxzBpycHYYe6DEHNpxCmby4P7X3xhoI0kpNHXA/K6OtA7BbVvJNGfDannRDvdSDYTU\"",

So at this point I'm wondering if 4 consecutive etags are not too conservative for this kind of resource ? How to make this process more robust?

[xens]

Ok problem is back today with the inconsistent etags :sweat_smile:, for now I've edited the consistencyCheck and compiled + deployed manually the provider following this document.

- const numConsistent = 4
+ const numConsistent = 2

and it's working like a charm.

[soloradish]

We encounter this problem as well. We just rm the user from the state and import it back.

[SamuZad]

This is an issue relating to the google APIs eventual consistency, and how the provider handles the checks for said eventual consistency

I significantly improved this in my fork: https://registry.terraform.io/providers/SamuZad/googleworkspace/latest/docs

The specific commit can be found here: https://github.com/SamuZad/terraform-provider-googleworkspace/commit/86270a6628f23caaf7455909d686d8e58912d074

Since implementing this, timing out during this operation has gone away completely, with no other problems cropping up 🙂

[red8888]

@SamuZad do you have a PR open with them to fix this? I'm consistently getting timeouts every time. Seems like this has been an issue for a long time.

[SamuZad]

@red8888 if you look at the commit history, this provider hasn't been updated in 2 years (outside of license declarations)

Which is why (as said above), I created a fork, that addresses a number of issues, including this one 🙂

I would love nothing more than for Hashicorp to be active with this provider again, alas it doesn't seem like it's in the cards. The moment there is some movement, I will open a PR here. Until then, I would suggest you use the forked version, linked above

[red8888]

@SamuZad were you able to open a PR for them? I don't currently see one.

Asking because I wanna help bug them and if we have a PR to point them to so we can just saw "merge this please" I'm hoping it will help push it forward

[SamuZad]

@red8888 I can only echo what I said above - this provider has not had active development done on it on YEARS. I will not be opening a PR until I see that someone from hashicorp is touching this repo, at which point I will be more than happy to. Use my fork if you want the timeout issue to be mitigated in the meantime