random string regenerates each time when override_special is defined

[deitch]

When I override the special characters to include the entire set of ascii chars (i.e. 0x00-0xff), it regenerates every single time.

Terraform Version

0.11.14

Affected Resource(s)

• random_string

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

resource "random_string" rander {
  length = 16
  override_special = "\x00\x01\x02\x03\x04\x05\x06\x07\x08\x09\x0a\x0b\x0c\x0d\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f\x20\x21\x22\x23\x24\x25\x26\x27\x28\x29\x2a\x2b\x2c\x2d\x2e\x2f\x30\x31\x32\x33\x34\x35\x36\x37\x38\x39\x3a\x3b\x3c\x3d\x3e\x3f\x40\x41\x42\x43\x44\x45\x46\x47\x48\x49\x4a\x4b\x4c\x4d\x4e\x4f\x50\x51\x52\x53\x54\x55\x56\x57\x58\x59\x5a\x5b\x5c\x5d\x5e\x5f\x60\x61\x62\x63\x64\x65\x66\x67\x68\x69\x6a\x6b\x6c\x6d\x6e\x6f\x70\x71\x72\x73\x74\x75\x76\x77\x78\x79\x7a\x7b\x7c\x7d\x7e\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff"
}

output rand {
  value = "${random_string.rander.result}"
}
output rand64 {
  value = "${base64encode(random_string.rander.result)}"
}

Debug Output

The relevant part is here:

2019/06/16 16:53:30 [TRACE] DiffTransformer: Module: DESTROY/CREATE: random_string.rander
  length:           "16" => "16"
  lower:            "true" => "true"
  min_lower:        "0" => "0"
  min_numeric:      "0" => "0"
  min_special:      "0" => "0"
  min_upper:        "0" => "0"
  number:           "true" => "true"
  override_special: "\x00\x01\x02\x03\x04\x05\x06\a\b\t\n\v\f\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\u007f��������������������������������������������������������������������������������������������������������������������������������" => "\x00\x01\x02\x03\x04\x05\x06\a\b\t\n\v\f\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\u007f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff" (forces new resource)
  result:           "9�\f2h��\x18Y�9�J�\v$" => "<computed>"
  special:          "true" => "true"
  upper:            "true" => "true"

it says that override_special is forcing a new resource.

Expected Behavior

What should have happened?

The input is unchanged, so the resource should be unchanged

Actual Behavior

What actually happened?

New resource

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Create main.tf in a dir with the above content
2. tf init .
3. tf apply .

Then run it again

[deitch]

It gets more interesting. I tried to fix it by setting:

 lifecycle {
   ignore_changes = ["override_special"]
 }

It actually does ignore the changes, but still changes the output. I suspect it has something to do with special characters.

tf apply -auto-approve .
random_string.rander: Creating...
  length:           "" => "16"
  lower:            "" => "true"
  min_lower:        "" => "0"
  min_numeric:      "" => "0"
  min_special:      "" => "0"
  min_upper:        "" => "0"
  number:           "" => "true"
  override_special: "" => "\x00\x01\x02\x03\x04\x05\x06\a\b\t\n\v\f\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~\u007f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff"
  result:           "" => "<computed>"
  special:          "" => "true"
  upper:            "" => "true"
random_string.rander: Creation complete after 0s (ID: none)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

len = 16
rand = ��3X�T��b uM�
rand64 = 9uyzM1itVI4D72IgdX9N4Q==
➜  tf tf apply -auto-approve .
random_string.rander: Refreshing state... (ID: none)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

len = 30
rand = ���3X�T��b uM�
rand64 = 77+977+977+9M1jvv71U77+9A++/vWIgdX9N77+9

The first time it has 1 added, the second time 0 changed, so it should have changed nothing. But the raw data is changed, and it is almost double the length. Interestingly, the data in the tfstate file changes from first run to second run.

First run:

                            "result": "\ufffd\u0002\ufffdO\ufffds\u0008\u0003x\ufffd\ufffd:\ufffd$\ufffds",

Second run:

                            "result": "�\u0002�O�s\u0008\u0003x��:�$�s",

I think it is doing some strange unicode encoding. I am not sure how to get around this.

All of this because of #63

[apparentlymart]

Hi @deitch,

Terraform strings are of unicode characters rather than bytes, so what you are trying to do here is not possible with Terraform strings. It would require some specialized handling of random bytes as you described in #63.

The weird behavior you saw with diffing is a result of the fact that 0.11 did not consistently validate input as UTF-8 and would e.g. allow hex character escapes to produce invalid UTF-8 which Terraform's internals would then corrupt by assuming valid UTF-8. In Terraform 0.12 I expect you'd see a different result... probably interpreting your override_special as a list of unicode characters at decoding time, and thus some confusion for the random_string resource because it is (incorrectly) assuming that override_special contains only ASCII characters.

I think there is still a bug to be fixed in random_string, though: it ought to be interpreting override_special as a set of unicode characters rather than bytes and then including those characters as a whole rather than picking individual bytes out of what are potentially multi-byte UTF-8 sequences. For consistency with how the rest of Terraform defines "unicode character" it should use textseg to do that splitting of the input.

Until that bug is fixed, only ASCII characters are usable in override_special.

[deitch]

Thanks for the explanation @apparentlymart. In the meantime I generated random_integer with count = 16, generated the output with join(“ “,...), and then some lovely shell magic with printf to turn them into actual bytes. Ugly, but it works.