search

hashicorp / terraform-cdk

Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform
https://www.terraform.io/cdktf
Mozilla Public License 2.0
4.86k stars 454 forks source link

splat expressions and Fn.values not working as expected #3145

Closed roflhouse closed 1 year ago

roflhouse commented 1 year ago

Expected Behavior

I have a use case where I need to lookup a list of objects at runtime and then extract a value out of the list. More specifically, I am using DataOpenstackNetworkingSubnetIdsV2 to run a search at runtime to find a list of ids which match a given criteria. Then I am using an iterator and DataOpenstackNetworkingSubnetV2 to lookup these ids to get the backing resources using a for_each statement. Both of these steps work fine and as expected. However, the issue comes in when I am trying to extract the data I need from the DataOpenstackNetworkingSubnetV2 namely the dynamic list of cidrs.

A simplified version of the above would look like:

list_of_ids = TerraformLocal(self.scope, f"list", ['f43657ef-a331-4108-883e-557fd4ff03d0', 'b13e710e-9276-493d-b561-6dc4cff2f5da', 'ad21a8c7-4ad4-4c28-8f18-4b87efaeec77'])
iterator = TerraformIterator.from_list(list=list_of_ids.as_list)
elements = DataOpenstackNetworkingSubnetV2(
     self.scope,
     f"elements",
     provider=self.provider,
     for_each=iterator,
     subnet_id=Token.as_string(iterator.value))
TerraformOutput(self.scope, f"outputs", value=elements)
TerraformOutput(self.scope, f"outputs2", value=elements.cidr)

The above TerraformOutput blocks generate the following terraform code:

"outputs": {
  "value": "${data.openstack_networking_subnet_v2.elements}"
},
"outputs2": {
  "value": "${data.openstack_networking_subnet_v2.elements.*.cidr}"
},

Actual Behavior

The first output works great and outputs all the objects correctly. The second output however isn't valid Terraform and results in the following error

Error: Unsupported attribute
│
│   on cdk.tf.json line 568, in output.outputs2:
│  568:       "value": "${data.openstack_networking_subnet_v2.elements.*.cidr}"
│
│ This object does not have an attribute named "cidr".

The problem is that data.openstack_networking_subnet_v2.elements is a map of maps. which looks like:

elements = { 
    "id1" = { "cidr" : ... },
    "id2" = { "cidr" : ... }
}

taking the splat expression of that object creates will just create a list of one element with then entire contents of elements in it. What I actually need is to generate the following command: "${values(data.openstack_networking_subnet_v2.elements).*.cidr}"

Steps to Reproduce

To reproduce this issue run the following CDKTF Python or modify it to use any resource instead of DataOpenstackNetworkingSubnetV2

list_of_ids = TerraformLocal(self.scope, f"{self.seed_name}_list", ['f43657ef-a331-4108-883e-557fd4ff03d0', 'b13e710e-9276-493d-b561-6dc4cff2f5da', 'ad21a8c7-4ad4-4c28-8f18-4b87efaeec77'])
iterator = TerraformIterator.from_list(list=list_of_ids.as_list)
elements = DataOpenstackNetworkingSubnetV2(
     self.scope,
     f"elements",
     provider=self.provider,
     for_each=iterator,
     subnet_id=Token.as_string(iterator.value))
TerraformOutput(self.scope, f"outputs", value=elements)
TerraformOutput(self.scope, f"outputs2", value=elements.cidr)

Versions

cdktf debug language: python cdktf-cli: 0.18.0 node: v18.17.1 cdktf: 0.17.3 constructs: 10.2.69 jsii: 1.85.0 terraform: 1.5.7 arch: x64 os: linux 5.15.90.1-microsoft-standard-WSL2 python: Python 3.10.12 pip: pip 23.2.1 from /home/nick/.local/lib/python3.10/site-packages/pip (python 3.10) pipenv: pipenv, version 2023.9.8

Providers

┌────────────────────────────────────────┬──────────────────┬─────────┬────────────┬───────────────────────────────┬─────────────────┐ │ Provider Name │ Provider Version │ CDKTF │ Constraint │ Package Name │ Package Version │ ├────────────────────────────────────────┼──────────────────┼─────────┼────────────┼───────────────────────────────┼─────────────────┤ │ terraform-provider-openstack/openstack │ 1.52.1 │ │ │ │ │ ├────────────────────────────────────────┼──────────────────┼─────────┼────────────┼───────────────────────────────┼─────────────────┤ │ google │ 4.75.0 │ ^0.17.0 │ │ cdktf-cdktf-provider-google │ 8.0.6 │ ├────────────────────────────────────────┼──────────────────┼─────────┼────────────┼───────────────────────────────┼─────────────────┤ │ local │ 2.4.0 │ ^0.17.0 │ │ cdktf-cdktf-provider-local │ 7.0.0 │ ├────────────────────────────────────────┼──────────────────┼─────────┼────────────┼───────────────────────────────┼─────────────────┤ │ template │ 2.2.0 │ ^0.17.0 │ │ cdktf-cdktf-provider-template │ 7.0.0 │ ├────────────────────────────────────────┼──────────────────┼─────────┼────────────┼───────────────────────────────┼─────────────────┤ │ vault │ 3.18.0 │ ^0.17.0 │ │ cdktf-cdktf-provider-vault │ 9.0.1 │ └────────────────────────────────────────┴──────────────────┴─────────┴────────────┴───────────────────────────────┴─────────────────┘

Gist

No response

Possible Solutions

I am not sure about the wider reaching impact of for_each in cdktf but it would seem that if you are making a splat expression from an object generated from a for_each you will always want to put it through Fn.values() first. As the generated resource will be of the type map of maps. So, one possibility is to insert that values call or add a function on the resource which will produce that. Someone with more experience with CDKTF would probably want to compare this case and solution with the count case. which i believe generates a list of maps. This solution does have the benifit of being the most intuitive to use. namely it matches the non-splat use case where if no for_each is used the user would need to write elements.cidr to get the single cidr value.

Another solution would be to change the way Fn.values() works. Currently, it returns string object which seems wrong. Instead, it could return a list object capable of generating splat statements. This solution seems the safest. it would require the user to understand why a splat statement on the for_each resources don't work but it would at least match the solution you would need to do in terraform.

Workarounds

The workaround here what i did was first to put the for_each object through Fn.values() and store it into a local variable. Then i used the local variable to generate the correct terraform code in another local variable. This local variable then contained the data i needed. This data was then used in an iterator object to be useful.

Also not that Fn.values() requires you to use the .fqn and failes if you try and use the local variable which also seems like a bug.

local_var = TerraformLocal(self.scope, f"public_cidrs", Fn.values(public_subnets.fqn))
# This is generating the splat statement "${local_var.public_cidrs.*.cidr}" which isn't suported except though this trickery
self.local_public_subnets_cidrs = TerraformLocal(self.scope, f"public_cidrs2", f"${{local.{local_var.friendly_unique_id}.*.cidr}}")

Anything Else?

Fn.values seems like it has a number of issues. First it returns a str object making it impossible to use with other Fn. functions like fn.element. It also doesn't allow the direct use of variable/resources and requires the user to use the .fqn parameter to workaround it.

One thing i have found that i often need when trying to use cdktf, especially when trying to work around issues like this, is access to the full terraform name unmodified. For example, when i wrote f"${{local.{local.friendly_unique_id}.*.cidr}}" i needed to hardcode in the local. and use the friendly_unique_id instead the resource just having an attribute which returns the terraform name. the fqn comes close but it i always end up with ${} around the name in bad places when i try and use it. Consider a raw_fqn parameter which never adds the ${} to the fqn.

References

No response

Help Wanted

Community Note

jsteinich commented 1 year ago

Thanks for the detailed writeup. Unfortunately this use case just isn't well supported at this time. Here are a couple existing issues that address different pieces:

It looks like Fn.values broke when switching to generated Terraform functions. That should definitely be corrected to return a list. It can be wrapped in Token.as_list as an alternative workaround.

xiehan commented 1 year ago

Closing as a duplicate of #2024 and #1641.

github-actions[bot] commented 11 months ago

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.