Open mforeman19 opened 1 year ago
Hi @mforeman19! Thanks for reporting this.
Looking at this part of your example:
concat(local.list_object_a, local.list_object_b)
These two lists seem to have different element types, so they cannot be concatenated together into a single list with a single element type. Looking at today's implementation of concat
I see that it does try to return a list, but it can do so only if all of the given lists have compatible element types.
Your example with list_of_objects_ternary_with_tolist
seems to confirm that's what's going on, because tolist
uses the same logic to try to determine a common element type as concat
does. The difference is just that concat
falls back on creating a tuple if it cannot create a list, whereas tolist
has no option but to fail because it would be invalid for it to return a non-list type.
I expect this would work if you made the two lists have unifyable object types, so that Terraform can find a suitable single object type to use as the type of the resulting list:
locals {
list_object_a_enabled = true
list_object_a = tolist([
{
id = "a"
enabled = true
expiration = null
nested = {
nested_id = 0
nested_class = "test"
}
}
])
list_object_b = tolist([
{
id = "b"
enabled = true
expiration = { days = 31 }
nested = null
}
])
list_of_objects_ternary = local.list_object_a_enabled == true ? concat(local.list_object_a, local.list_object_b) : local.list_object_b
list_of_objects_ternary_with_tolist = local.list_object_a_enabled == true ? tolist(concat(local.list_object_a, local.list_object_b)) : local.list_object_b
list_of_objects_ternary_false = local.list_object_a_enabled == false ? concat(local.list_object_a, local.list_object_b) : local.list_object_b
}
The extra null
attributes I added above should allow Terraform to determine that the result type of concat
would be the following list type, after noticing that null
is a valid value of any type:
list(object({
id = string
enabled = bool
expiration = object({
days = number
})
nested = object({
nested_id = number
nested_class = test
})
}))
Can you give that variant a try and see if you get the behavior you wanted?
Hey @apparentlymart, I gave that a go and got this error back:
│ Error: Error in function call
│
│ on main.tf line 128, in locals:
│ 128: list_of_objects_ternary = local.list_object_a_enabled == true ? concat(local.list_object_a, local.list_object_b) : local.list_object_b
│ ├────────────────
│ │ while calling concat(seqs...)
│ │ local.list_object_a is list of object with 1 element
│ │ local.list_object_b is list of object with 1 element
│
│ Call to function "concat" failed: panic in function implementation: inconsistent list element types (cty.Object(map[string]cty.Type{"enabled":cty.Bool,
│ "expiration":cty.DynamicPseudoType, "id":cty.String, "nested":cty.Object(map[string]cty.Type{"nested_class":cty.String, "nested_id":cty.Number})}) then
│ cty.Object(map[string]cty.Type{"enabled":cty.Bool, "expiration":cty.Object(map[string]cty.Type{"days":cty.Number}), "id":cty.String, "nested":cty.DynamicPseudoType}))
│ goroutine 798 [running]:
│ runtime/debug.Stack()
│ /opt/hostedtoolcache/go/1.19.6/x64/src/runtime/debug/stack.go:24 +0x65
│ github.com/zclconf/go-cty/cty/function.errorForPanic(...)
│ /home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/function/error.go:44
│ github.com/zclconf/go-cty/cty/function.Function.Call.func1()
│ /home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/function/function.go:294 +0x9f
│ panic({0x260daa0, 0xc0032750a0})
│ /opt/hostedtoolcache/go/1.19.6/x64/src/runtime/panic.go:884 +0x212
│ github.com/zclconf/go-cty/cty.ListVal({0xc001bb92c0, 0x2, 0x2511ae0?})
│ /home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/value_init.go:166 +0x42e
│ github.com/zclconf/go-cty/cty/function/stdlib.glob..func79({0xc001bb90c0, 0x2, 0x2511ae0?}, {{0x3072158?, 0xc003274ae0?}})
│ /home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/function/stdlib/sequence.go:108 +0x87f
│ github.com/zclconf/go-cty/cty/function.Function.Call({0x3072158?}, {0xc001bb90c0?, 0x2, 0x2})
│ /home/runner/go/pkg/mod/github.com/zclconf/go-cty@v1.12.1/cty/function/function.go:298 +0x2a4
│ github.com/hashicorp/hcl/v2/hclsyntax.(*FunctionCallExpr).Value(0xc0001b6690, 0xc0024204b0)
│ /home/runner/go/pkg/mod/github.com/hashicorp/hcl/v2@v2.16.2/hclsyntax/expression.go:456 +0x1c85
│ github.com/hashicorp/hcl/v2/hclsyntax.(*ConditionalExpr).Value(0xc0001dd340, 0xc0024204b0)
│ /home/runner/go/pkg/mod/github.com/hashicorp/hcl/v2@v2.16.2/hclsyntax/expression.go:634 +0x4a
│ github.com/hashicorp/terraform/internal/lang.(*Scope).EvalExpr(0xc000ff0a20, {0x3070ec0?, 0xc0001dd340}, {{0x3072120?, 0x4518310?}})
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/lang/eval.go:171
│ +0x148
│ github.com/hashicorp/terraform/internal/terraform.(*BuiltinEvalContext).EvaluateExpr(0x3070ec0?, {0x3070ec0, 0xc0001dd340}, {{0x3072120?, 0x4518310?}}, {0x0?, 0x0?})
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/terraform/eval_context_builtin.go:283
│ +0xc5
│ github.com/hashicorp/terraform/internal/terraform.(*NodeLocal).Execute(0x0?, {0x308a1d8, 0xc000f8e0e0}, 0x0?)
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/terraform/node_local.go:154
│ +0x5aa
│ github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0022fb0e0, {0x308a1d8, 0xc000f8e0e0}, {0x15bd63baac0, 0xc000fe2cf0})
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/terraform/graph_walk_context.go:136
│ +0xc2
│ github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x28397c0, 0xc000fe2cf0})
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/terraform/graph.go:75
│ +0x315
│ github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000ad64e0, {0x28397c0, 0xc000fe2cf0}, 0xc000162040)
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/dag/walk.go:381
│ +0x2f6
│ created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update
│ /home/runner/work/_temp/actions-go-build/0.1.7/b712d316a0484806ec3fa2c744c064f8874ceff1/verification/ffa6843d8160ea6b18c9f31a4cfafe4e85dc17655c94b0feed83b13a6dcf747a/cache/source/terraform/terraform/6c2c6cfa1b55bd6ff4cf4e26ef86d8d7ab0465ec/internal/dag/walk.go:304
│ +0xf65
│ .
Where this is the input:
list_object_a_enabled = true
list_object_a = tolist([
{
id = "a"
enabled = true
nested = {
nested_id = 0
nested_class = "test"
}
expiration = null
}
])
list_object_b = tolist([
{
id = "b"
enabled = true
nested = null
expiration = { days = 31 }
}
])
concat seems to work fine if you have non-complex types within the objects in comparison. Here's an example where one object has attributes that the other does not:
list_object_a_enabled = true
list_object_a = tolist([
{
id = "a"
enabled = true
in_a = true
}
])
list_object_b = tolist([
{
id = "b"
enabled = true
in_b = "in_b"
}
])
list_of_objects_ternary = local.list_object_a_enabled == true ? concat(local.list_object_a, local.list_object_b) : local.list_object_b
Where the output from grabbing type is:
> type(local.list_of_objects_ternary)
list(map(string))
Thanks for trying that @mforeman19!
Based on that "panic" it seems like we've found a bug in the implementation of the concat
function where it's trying to behave as you wanted and as I claimed it should -- it tried to construct a list value to return -- but it's done so incorrectly and so it crashed at the last moment trying to assemble the resulting list.
Since we're already in a bug report issue anyway I think we should consider this issue to represent fixing that crash bug so the function will correctly return the list it was trying to return. The implementation of this function is actually in a third-party library upstream so we'll need to contribute a fix upstream to that, but we can still use this issue to track upgrading that dependency in Terraform once fixed so that Terraform will benefit from the fix.
Based on what was reported in the error message, my guess as to the cause is that it seems to be trying to build the resulting list from the values given directly as arguments, rather than using the results of converting the arguments to match the selected result type. I'm basing this on the fact that the two types mentioned in the error message still have cty.DynamicPseudoType
, which is the placeholder used to represent the fact that null
can be of any type, but by the time it's constructing its result those should have already been converted to the concrete result element type.
Sounds good, appreciate the quick response @apparentlymart. If it helps for context, most of the testing I was doing was via running terraform console on a local plan (not stored in like s3).
I am interested to work on this issue. Can any one help me from where to start ?
Hi @jay-laitmatus, please see https://github.com/hashicorp/terraform/blob/main/.github/CONTRIBUTING.md#contributing-a-pull-request. To set expectations, this is unlikely to be a good first issue to learn how to contribute to Terraform.
My relevant terraform code:
variables.tf
:
variable "home_block_device" {
description = "Settings for the home block device"
type = object({
device_name = string,
delete_on_termination = optional(bool, false),
encrypted = optional(bool, false),
volume_type = optional(string, "gp3"),
iops = optional(number, 3000),
throughput = optional(number, 125), # MiB/s
volume_size = number,
tags = optional(map(string))
})
default = null
}
variable "secondary_block_devices" { description = "Settings for secondary block devices" type = list(object({ device_name = string, delete_on_termination = optional(bool, false), encrypted = optional(bool, false), volume_type = optional(string, "gp3"), iops = optional(number, 3000), throughput = optional(number, 125), # MiB/s volume_size = number, tags = optional(map(string)) })) default = [] }
* `locals.tf`:
```hcl
locals {
home_block_device = var.home_block_device
secondary_block_devices = var.secondary_block_devices
additional_block_devices = local.home_block_device != null ? tolist(concat([local.home_block_device], local.secondary_block_devices)) : local.secondary_block_devices
}
This seems like another case for this bug, I see the error output:
╷
│ Error: Inconsistent conditional result types
│
│ on ec2_instance/locals.tf line 50, in locals:
│ 50: additional_block_devices = local.home_block_device != null ? tolist(concat([local.home_block_device], local.secondary_block_devices)) : local.secondary_block_devices
│ ├────────────────
│ │ local.home_block_device is null
│ │ local.secondary_block_devices is empty list of object
│
│ The true and false result expressions must have consistent types. Mismatched list element types: The 'false' value
│ includes object attribute "delete_on_termination", which is absent in the 'true' value.
╵
I am left scratching my head wondering how the 'false' value can include the object attribute "delete_on_termination", when the value itself is shown as being an "empty list of object"?!?!
Additionally as local.secondary_block_devices
which is the false
value also appears as part of the possible true
value, I would expect Terraform to correctly calculate the result type.
Tested on Terraform Version: 1.5.7 and 1.6.1
It seems to me that there is a bug in the Type System of Terraform here. If we look at how the types coalesce in another functional language (e.g. Scala) with code which is exactly equivalent to the Terraform code:
case class BlockDevice(device_name: String, delete_on_termination: Boolean, encrypted: Boolean, volume_type: String, iops: Number, throughput: Number, volume_size: Number, tags: Map[String, String])
val home_block_device : BlockDevice = null
val secondary_block_devices: List[BlockDevice] = List.empty[BlockDevice]
val result = if (home_block_device != null) List.concat(List(home_block_device), secondary_block_devices) else secondary_block_devices
This correctly computes the result: val result: List[BlockDevice] = List()
My guess would be that Terraform is loosing the object type information for the variable home_block_device
when it is set to a null
value.
Ultimately I just want a List containing both (a) var.home_block_device
(if it is present), and (b) any entries from var.secondary_block_devices
. Is there some other workaround that I can use for this perhaps?
An empty list can still have an element type, which is the type that its elements would have if it were not empty. For example, if you declare an input variable of type list(string)
and then the caller assigns []
to it then the result will be an empty list that knows its element type is string.
Not all empty lists have a known element type -- Terraform also supports "empty list of unknown element type" for situations like tolist([])
-- but where possible Terraform prefers to track the element type of an empty list because it can help catch bugs earlier in the workflow -- that is, during validate or plan instead of only during apply.
In the example in the comment directly above this one, I assume that Terraform was able to infer the element type of local.secondary_block_devices
and so it can see that it's inconsistent. If so, that seems like correct behavior rather than a bug, though of course seeing more context around how those source values are defined might suggest that Terraform made an incorrect conclusion about the empty list's element type, in which case that would be a bug.
@apparentlymart Thanks for the quick follow-up. As suggested, I have now added the relevant Terraform code into my report above.
Hello @mforeman19 and others,
here's the hackishism I would shamefully use in this case:
locals {
list_object_a_enabled = true
list_object_a = tolist([
{
id = "a"
enabled = true
nested = {
nested_id = 0
nested_class = "test"
}
}
])
list_object_b = tolist([
{
id = "b"
enabled = true
expiration = { days = 31 }
}
])
list_of_objects_ternary = try(local.list_object_a_enabled ? flatten([local.list_object_a, local.list_object_b]) : tolist(false), local.list_object_b)
list_of_objects_ternary_with_tolist = try(local.list_object_a_enabled ? tolist(flatten([local.list_object_a, local.list_object_b])) : tolist(false), local.list_object_b)
list_of_objects_ternary_false = try(!local.list_object_a_enabled ? flatten([local.list_object_a, local.list_object_b]) : tolist(false), local.list_object_b)
}
namely, flatten overcomes the concat issue, while try and forced error in ternary overcomes the ternary consistent type requirement.
result:
terraform console
> local.list_of_objects_ternary
[
{
"enabled" = true
"id" = "a"
"nested" = {
"nested_class" = "test"
"nested_id" = 0
}
},
{
"enabled" = true
"expiration" = {
"days" = 31
}
"id" = "b"
},
]
> local.list_of_objects_ternary_with_tolist
tolist([
{
"enabled" = true
"expiration" = {
"days" = 31
}
"id" = "b"
},
])
> local.list_of_objects_ternary_false
tolist([
{
"enabled" = true
"expiration" = {
"days" = 31
}
"id" = "b"
},
])
>
I ended up just encoding the lists as Json and concatting the json lists together.
Hi all... I am too facing a similar issue and I belive you can help.
Below is a piece of my code that I am trying out to setup an ALB for my EC2.
#Create an EC2 Instance
resource "aws_instance" "terra_prj1_ec2" {
ami = "ami-0014ce3e52359afbd"
instance_type = "t3.micro"
availability_zone = "eu-north-1a"
count = 2
key_name = "main-key"
iam_instance_profile = aws_iam_instance_profile.terra_prj1_instance_profile.name
user_data = file("userdata.sh")
network_interface {
device_index = 0
network_interface_id = aws_network_interface.terra_prj1_sec_network_interface.id
}
}
locals {
ec2_instace_ids = join("", aws_instance.terra_prj1_ec2.*.id)
}
#Create a security group for load balancer
resource "aws_security_group" "terra_prj1_alb_sec_grp" {
name = "allow_terra_web_traffic"
description = "Allow web inbound traffic"
vpc_id = aws_vpc.terra_prj1.id
ingress {
description = "HTTP"
from_port = 80
to_port = 80
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
ipv6_cidr_blocks = ["::/0"]
}
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
ipv6_cidr_blocks = ["::/0"]
}
}
#Create a security group rule for EC2
resource "aws_security_group_rule" "terra_prj1_alb_sec_grp_rule" {
type = "ingress"
from_port = 8080
to_port = 8080
protocol = "tcp"
security_group_id = aws_security_group.terra_prj1_sec_grp.id
source_security_group_id = aws_security_group.terra_prj1_alb_sec_grp.id
}
#Create a target group for load balancer
resource "aws_lb_target_group" "terra_prj1_alb_target_grp" {
name = "terra-prj1-alb-target-grp"
port = 8080
protocol = "HTTP"
vpc_id = aws_vpc.terra_prj1.id
load_balancing_algorithm_type = "round_robin"
health_check {
enabled = true
port = 8081
interval = 30
protocol = "HTTP"
path = "/health"
matcher = 200
healthy_threshold = 3
unhealthy_threshold = 3
}
}
#Create target group attachment
resource "aws_lb_target_group_attachment" "terra_prj1_alb_target_grp_attachment" {
for_each = toset(local.ec2_instace_ids)
target_group_arn = aws_lb_target_group.terra_prj1_alb_target_grp.arn
target_id = each.value
depends_on = [aws_lb_target_group.terra_prj1_alb_target_grp]
port = 8080
}
#Create a load balancer
resource "aws_lb" "terra_prj1_alb" {
name = "terra-prj1-alb"
internal = false
load_balancer_type = "application"
security_groups = [aws_security_group.terra_prj1_alb_sec_grp.id]
subnets = [aws_subnet.terra_prj1_subnet.id]
}
#Create a listener for load balancer
resource "aws_lb_listener" "terra_prj1_alb_listener" {
load_balancer_arn = aws_lb.terra_prj1_alb.arn
port = 80
protocol = "HTTP"
default_action {
type = "forward"
target_group_arn = aws_lb_target_group.terra_prj1_alb_target_grp.arn
}
}
I am facing the below error while running the code.
│ Error: Invalid function argument
│
│ on Main.tf line 298, in resource "aws_lb_target_group_attachment" "terra_prj1_alb_target_grp_attachment":
│ 298: for_each = toset(local.ec2_instace_ids)
│ ├────────────────
│ │ while calling toset(v)
│ │ local.ec2_instace_ids is a string
│
│ Invalid value for "v" parameter: cannot convert string to set of any single type.
╵
tried multiple sources to fix hence no luck so far. Appreciate if anyone can have a look
I ended up just encoding the lists as Json and concatting the json lists together.
Freaking genius!!!
I myself is dealing with quite nested and complex list of objects often, and I hit the "types are not the same" So many times where the compiler simply cannot see far enough "down" The nesting to determine 2 different lists actually contain the exact same type of objects, BUT the lists are not the same length.
By simply converting both sides of true & false to string via jsonencode, we simply tell the compiler, HEY its just strings, calm down!! Awesome mate :)
Terraform Version
Terraform Configuration Files
Debug Output
Expected Behavior
If I concat lists, I expect a list to be returned, not a tuple. This seems to result from complex objects (like nested maps) being involved in the concatenation. I haven't noticed this if the lists contained string or bool or int. Perhaps this has already been found or the behavior is expected.
Actual Behavior
See errors
Steps to Reproduce
Attempt to concat two lists that contain complex objects (like a map)
Additional Context
It'd be useful to concat these types of objects together, particularly in use for modules. I have lists containing complex object variables I'm trying to pass into modules and then perform some modifications on top of those variables if certain conditions are met. Sometimes these variables have a certain nested map in them, sometimes they don't. Sometimes it might just be string fields instead of string fields + map objects within an object in the list.
References
No response