Closed prehor closed 2 years ago
I'd really like to have this option too. Any plans of prioritizing this enahancement ?
is there a workaround for this? I found myself using type=any
@rverma-nikiai No, there is not currently a workaround for this.
The best thing we can do is define the optional argument as such (where foobar
is the optional property):
modules/myfoo/main.tf
variable "foo" {
type = object({
name = string
id = number
foobar = string # or bool, number, etc...
})
}
resource "null_resource" {
triggers = {
some_name = var.foo.name
some_id = var.foo.id
some_foobar = var.foo.foobar
}
}
... and then call it, explicitly setting the optional properties to null
main.tf
module "potato" {
source = "./modules/myfoo"
foo = {
name = "something-cool-is-afoot"
id = 1234567890
foobar = null
}
}
This allows you to do normal stuff like set defaults using a ternary check, eg.:
modules/myfoo/main.tf
locals {
foobar_default = var.foo.foobar == null ? "default value for foo.foobar" : null
}
...
Along the same lines, it would be useful to be able to declare partial defaults for complex types
For instance, something like
variable "with_default_path" {
type = object({
id = string
path = string
})
default = {
path = "/"
}
}
Hi folks! This is a great feature request, and getting use-cases is absolutely helpful, but so you know it best to react to the original issue comment with 👍which we can and do report on during prioritization. Thank you :)
Along the same lines, it would be useful to be able to declare partial defaults for complex types
For instance, something like
variable "with_default_path" { type = object({ id = string path = string }) default = { path = "/" } }
Absolutely, this should be part of the original post as well
👍 Optional values 👍 Partial default values
@solsglasses - please add reactions to comments, pull requests, or issues instead of creating a new comment simply to show agreement with someone. Thanks for helping to keep the community awesome, and the notification feed of -- well, everyone -- free of fluff!
I'm not sure of the internals, but this may not be that difficult. Since type and defaults are already well defined, the parser would just need to allow both of them and then merge the defaults with the provided variable before validation.
My proposal mostly dealt with optional values inside the type definition, not the argument itself being optional. The might get a little trickier. Having the argument be optional is going to lead to a lot of lines like:
for_each = var.network_rules == null ? [] : list(var.network_rules)
It may make more sense for the default
value to match the type
definition. So a list(object(...))
would return an empty list. This way you can let the parser handle the null check and you can be guaranteed to get the proper type
for iteration. This is assuming that an empty list
is valid for the type
definition of list(object(...))
.
It could reduce a bunch of boilerplate code if there was an optional
flag in the variable definition.
Update: I created #22449 to track this.
One more proposal to add to the mix:
object({
<KEY> = <TYPE | { [type = <TYPE>,][ default = ...] }>
})
I think would allow more readable complex object definitions in cases where defaults could appear at any level, and would have parity with how defaults in variable
blocks are defined. For a contrived example:
variable "pipeline" {
type = object({
name = string
pipeline_type = {
type = string // inferred from default & can be omitted
default = "STANDARD"
}
stages = {
type = list(object({
name = string
cmd = string
must_pass = { default = false }
}
}))
default = [
{
name = "build"
cmd = "make all"
must_pass = true
},
{
name = "deploy"
cmd = "make deploy"
},
]
}
})
default = {
name = "standard_build_pipeline"
// everything else is taken from the type defaults
}
}
(the use-case for having both type
and default
in the block would be for when the type can't be inferred from the default, such as when default = null
).
For what it's worth, even the minimal behavior (accepting objects with omitted values and setting the omitted values to null) would be a major usability improvement. Defaults could be implemented within modules themselves.
Very needed possibility for maps and objects types.
Being able to set a default = null
type I think would allow modules to set object type constraints for input variables or resources that have optional attributes, while still being able to set some type/object constraints.
My personal vote is also to add support for default values, rather than support for marking some fields as optional.
Defaults imply optionality, so you get optionals for free by supporting defaults. OTOH, if we add optional(...)
support, and then later add defaults, then we'll have two different ways of defining optionality (specifying a default vs specifying optional(...)
, the latter being equivalent to specifying a null
default), which I think would be less preferred.
Definitely @atavakoliyext proposal is best.
@prehor thefotios proposal:
variable "with_default_path" {
type = object({
id = string
path = string
})
default = {
path = "/"
}
}
Why we cannot use it:
variable "with_default_path" {
type = list(object({
id = string
path = string
}))
default = [{
path = "/"
}] # ???
# or how?
# this doesn't look reasonable
}
Actually there are workarounds for optional arguments in object variables. The tricky part is still on the default value of the variable and using local variables. Use the locals variable to verify the optional parameter in "network_rules". Put it into null if not existing. The other resources would be able to refer to those local variables.
variable "network_rules" {
default = {
bypass = null,
ip_rules = null,
virtual_network_subnet_ids = null
}
type = object({
bypass = list(string)
ip_rules = list(string)
virtual_network_subnet_ids = list(string)
})
}
locals {
network_rules = lookup(var.network_rules, "bypass", "no_bypass") == "no_bypass" ? null : var.network_rules,
ip_rules = lookup(var.ip_rules, "ip_rules", "no_ip_rules") == "no_ip_rules" ? null : var.network_rules,
virtual_network_subnet_ids = lookup(var.virtual_network_subnet_ids, "virtual_network_subnet_ids", "no_virtual_network_subnet_ids") == "no_virtual_network_subnet_ids" ? null : var.network_rules
}
@lemontreeran This does not work. I cleaned up your example:
variable "network_rules" {
default = {
bypass = null,
ip_rules = null,
virtual_network_subnet_ids = null
}
type = object({
bypass = list(string)
ip_rules = list(string)
virtual_network_subnet_ids = list(string)
})
}
locals {
network_rules = lookup(var.network_rules, "bypass", "no_bypass") == "no_bypass" ? null : var.network_rules
ip_rules = lookup(var.network_rules, "ip_rules", "no_ip_rules") == "no_ip_rules" ? null : var.network_rules
virtual_network_subnet_ids = lookup(var.network_rules, "virtual_network_subnet_ids", "no_virtual_network_subnet_ids") == "no_virtual_network_subnet_ids" ? null : var.network_rules
}
And provided a terraform.tfvars like so:
network_rules = {
bypass = ["nope"]
}
The result:
> $ terraform plan
Error: Invalid value for input variable
on terraform.tfvars line 1:
1: network_rules = {
2: bypass = ["nope"]
3: }
The given value is not valid for variable "network_rules": attributes
"ip_rules" and "virtual_network_subnet_ids" are required.
You cannot have optional elements of a well-defined variable object.
@lemontreeran This does not work. I cleaned up your example:
variable "network_rules" { default = { bypass = null, ip_rules = null, virtual_network_subnet_ids = null } type = object({ bypass = list(string) ip_rules = list(string) virtual_network_subnet_ids = list(string) }) } locals { network_rules = lookup(var.network_rules, "bypass", "no_bypass") == "no_bypass" ? null : var.network_rules ip_rules = lookup(var.network_rules, "ip_rules", "no_ip_rules") == "no_ip_rules" ? null : var.network_rules virtual_network_subnet_ids = lookup(var.network_rules, "virtual_network_subnet_ids", "no_virtual_network_subnet_ids") == "no_virtual_network_subnet_ids" ? null : var.network_rules }
And provided a terraform.tfvars like so:
network_rules = { bypass = ["nope"] }
The result:
> $ terraform plan Error: Invalid value for input variable on terraform.tfvars line 1: 1: network_rules = { 2: bypass = ["nope"] 3: } The given value is not valid for variable "network_rules": attributes "ip_rules" and "virtual_network_subnet_ids" are required.
You cannot have optional elements of a well-defined variable object.
Try adding the null value for those vars. I know it is ugly......
network_rules = {
bypass = ["nope"],
ip_rules = null,
virtual_network_subnet_ids = null
}
Try adding the null value for those vars. I know it is ugly......
network_rules = { bypass = ["nope"], ip_rules = null, virtual_network_subnet_ids = null }
It's not that it is ugly, it is not optional if you have to supply a value, which is the whole point of this issue.
@grimm26 Yes, that is why the issue is still open. As @lemontreeran noted, and as I wrote earlier in the thread, providing null
values for the "optional" parameters is the only way to circumvent the issue.
Or following the first post, I'd like to pass default values like this:
variable "network_rules" {
default = null
type = object({
bypass = optional(list(string), ["teste"])
ip_rules = optional(list(string), null)
virtual_network_subnet_ids = optional(list(string), [])
enabled = optional(bool, true)
})
}
The optional
could work like the lookup
function (without the first map
parameter, off course), receiving the type on the first parameter and the default value on the second one.
Where we have a map or list of map the solution should include the capability for the map key to be absent.
type = list(object({
foo = optional(string,"foo_default")
bar = optional(string)
}))
so if bar is not provided by the caller the key would be completely absent.
this way you can use the absence of the key to infer some more complex default action rather than having to use a rouge value for default.
Any idea if/when this will be addresed?
This is the most upvoted issue, can we get an update, please... @mildwonkey, @apparentlymart
Without optional param in objects, it really limits the level of abstraction that can be designed in a module.
I'm using the null pattern in lists with objects for dynamic blocks for CloudFront origins. And optional parameter would be nice to skip defining the http_port attribute when using https-only protocol policy. But as thinking about this you will fast come to the thought of „conditional optional parameters“ (e.g. http_port is only optional when corresponding protocol_policy parameters are set). This is imho against the declarative approach…
Wow, more than a year and 410 upvotes in the original post. @mildwonkey is there any chance this is gonna land in the roadmap at some point?
It is unfortunately easiest to work around this right now by using type = map(any)
and lookup(object, "key", "default")
, so long as you can put up with the terrible feeling of guilt it gives you.
I'd also like for this feature to be implemented. Here is what I did to get around not having optional arguments using merge and a bunch of for loops for AWS.
I can now use the variable var.websites in multiple resources to make sure the dns, SSL certificate, and cloudfront config all get created off of one variable while still allowing me the flexibility to have each website have slightly different parameters.
variable "config" {
type = any
}
config = {
"site1.example.com" = {
enabled = true,
}
"site2.example.com" = {
enabled = true,
}
}
locals {
default_config = {
setting = null,
enabled = false,
}
config_merged = {
for key, value in var.config : key => merge(local.default_config, value)
}
cf_config = {
for key, value in local.config_merged :
key => value
if value.enabled == true
}
}
resource "aws_cloudfront_distribution" "config" {
for_each = local.cf_config
comment = each.key
aliases = each.value.aliases != null ? each.value.aliases : (length(regexall("\\.", each.key)) <= 1 ?
["www.${replace(each.key, "/.*?([\\w]+\\.[\\w]+)$/", "$1")}"] : ["${each.key}"])
enabled = true
{...everything else...}
}
Something like this can be used as a replacement. Supports required, optional, nested/complex structures and type-checking. Not very readable though.
variable "config" {
type = any
}
locals {
config = {
required = tostring(var.config.required)
optional1 = tostring(try(var.config.optional1, "default1"))
optional2 = [for v in try(var.config.optional2, ["default2", "default3"]): tostring(v)]
sub = tomap(merge(
{
optional4 = "default4"
},
try(var.config.sub, {})
))
}
}
output "config" {
value = local.config
}
with input
config = {
required = "xxx"
optional1 = "yyy"
sub = {
optional4 = "zzz"
}
}
This feature would be great indeed. I my case, I'm passing an object that should have some default attributes. But since it's not possible to do it, I need to inject some default values when I call the module.
Example :
variable environments {
description = "List of the environments attached to this site"
type = map(object({
ecs_cluster_name = string
primary_url = string
additional_urls = list(string)
}))
}
When I call my module, I have to set additional_urls = []
Which works but is quite ugly, because most of my module invocations will have this empty value.
Ideally, it should be possible to set a default value for addition_urls, just like for other flat variables using the default
keyword.
Cheers
In addition to what others have said, I believe in the end of day what most people want is a way of defining complex blocks the same way native resources do. For example, in aws_instance
one can define a network_interface
block:
network_interface {
network_interface_id = "${aws_network_interface.foo.id}"
device_index = 0
}
Reading the docs for this block and others, we can see that:
It would be great to be able to define similar interfaces in my modules.
My scenario is described here - https://stackoverflow.com/questions/62105031/how-to-extend-terraform-module-input-variable-schema-without-breaking-existing-c?noredirect=1#comment109844759_62105031
Even without explicit default value, i.e. using fixed default values, like:
false
for bool
""
for string
This would be tremendously useful.
I do not really get it. This issue is open for 1.5 years. It is such a useful feature. It must be very difficult to implement, otherwise it would have been already. Can any one from the dev team provide some kind of a reason why this is so difficult to implement?
And the regression risk is minimal. At the very worst, the code that fails because of missing property would start working. So, there is no risk that an already working code would break.
@MarkKharitonov here is something I got from the product managers:
I’m saddened that this feature has left our community frustrated. Daniel and I only recently began with the team and have been working through the backlog of issues and concerns.
For 0.13 our goal was to tackle module expansion (count, for_each and depends_on) and deliver a significant update without the pain that many associated with our 0.12 release. The module-expansion work, when all related feature requests were combined, outweighed any other feature in terms of demand. We also felt this work delivered the broadest value to our community, which includes many who do not contribute or participate in GitHub.
By that reasoning however, #19898 is a strong possibility in our next major release cycle.
One of the highest level goals we’ll be working on, over the next year, is increasing our communication with the community about what we’re doing and not doing. We will also engage in targeted discussions about what would be of most value. Sometimes we’ll do that in GitHub issues, and other times we’ll engage elsewhere. This turnaround won’t be overnight, but we are already hard at work.
Petros Kolyvas <petros@hashicorp.com>
Daniel Dreier <ddreier@hashicorp.com>
@heldersepu - Thanks.
I would argue this particular feature could make life easier for commercial clients as well. Suppose HashiCorp wants to extend the schema of the modules they provide. They would face with the same dilemma. Either create new module as an almost replica of the old, but with the new schema or just change the schema and break all the existing code. Breaking existing code how it works today, I guess, If I am to judge from the terraform azurerm provider upgrade from 1.44 to 2.0. Sometimes one must do it, but sometimes it can be avoided with little things like reasonable default values. Anyway, I will wait, because what other choice is there, right?
I'd really like to have this option too. Any plans of prioritizing this enhancement ?
There is some workaround, using merge. See this blog: https://binx.io/blog/2020/01/02/module-parameter-defaults-with-the-terraform-object-type/
Im very new to using terraform and this is one of the first things I ran into.
Since I have not used it before I may be missing something obvious, but is there any reason why we can't have named types? (Which could have there own defaults)
The first thing I attempted was something along the lines of:
type "string_list_with_default" {
type = list(string)
default = "[]"
}
variable "network_rules" {
default = null
type = object({
bypass = string_list_with_default
ip_rules = string_list_with_default
virtual_network_subnet_ids = string_list_with_default
})
}
(Like I said, Im new to this so ignore the syntax part and focus on the "named custom types" which can have defaults part.)
With 0.13 announced, and with custom variable validation moving forward - I'm wondering if the near-to-mid-term "best practice" should be:
In absense of support for "optional" in the nearterm, does the above give us the desired behavior or is there anything additional I am missing from functional requirements?
With 0.13 announced, and with custom variable validation moving forward - I'm wondering if the near-to-mid-term "best practice" should be:
- declare these complex types as maps instead of declaring them as objects.
- use validation rules to ensure: a. required keys are present b. any keys provided which are neither required nor optional would raise "an error indicating unknown key"
In absense of support for "optional" in the nearterm, does the above give us the desired behavior or is there anything additional I am missing from functional requirements?
The problem with using a map instead of a well define object is with a map all of the values have to be the same type.
For example if I declare the type as map(string) all of the map values have be to strings. If I declare the type as map or map(any). The values can be of any type, but all of the values in the map have to be the same type. Where a well defined object I can have sting keys, but each key may be a different type.
@ritzz32 - Got it - and yes, I think you are right. My own use case also requires that some of the object's properties be strings (for instance) and some be lists of strings. So I think we are still blocked, as you say:
1:
The problem with using a map instead of a well define object is with a map all of the values have to be the same type.
2:
If I declare the type as map or map(any). The values can be of any type, but all of the values in the map have to be the same type.
Im currently doing more or less as @aaronsteers suggested above with variable validation (try()
), but have set type = any
directly for my complex object variables.. so not even declaring it as a map and thus getting around the "all having to be same type problem" @ritzz32 brought up.. it works for now, but does not feel very good..
@tomasbackman - thanks very much for this info. So, basically we don't set it up as a dict - and it'll be interpreted perhaps as an object internally (of parent class 'any'). Very clever!
I have been struck by another related issue which I think should be combined with this one. Specifically, the problem with checking the presence of a nullable object.
Related problem of is-missing checks not having deterministic results
Take this example:
variable "s3_logging_bucket" {
description = "Optional. The bucket where logs should be written. If omitted, logs will not be preserved"
type = string
default = null
}
resource "s3_bucket_policy" "my_bucket_policy" {
count = var.s3_logging_bucket == null ? 0 : 1
// ...
}
Seems straightforward enough: it works fine if you pass null
, it works fine if you pass nothing at all, and it works fine if you pass a string. However, if you pass in the output of another module, for instance if you pass s3_logging_bucket = module.data_lake.logging_bucket
it will fail during static analysis (during plan
) because it cannot detect what count
will be ahead of time, and you'll get the error that looks like "value of count
cannot be computed".
This error makes sense at first - terraform needs to deterministically know how many resources it's creating/tracking, right? - except and until you realize that the module is promising you it will never output a non-null bucket name; then, this becomes a very frustrating issue with no good solution. The only solution I've found is to add another boolean value like save_logging_to_s3
that is redundant, but deterministic. It would be preferrable to be able to declare a module's outputs as 'not null' so simple existence checks can still be calculated even though the exact output value is still unknown.
The unified proposal
We could introduce optional([type])
and required([type])
- with a smart default to one or the other depending upon context (probably defaulting to optional for every case except within Object() definitions where the default behavior is currently similar to required
).
Examples:
optional(str)
contains a string or the value null
str
- same as optional(str)
required(str)
is always a valid stringrequired(bool)
contains either true
or false
(but never null
)required(list(str))
which contains a list of zero or more strings (but will never fail when evaluating length()
)Tying it all back together, when the optional() pattern is applied to Object members, this also gives us the language necessary to permit missing values while declaring the object.
In my count
example above, we solve it simply by declaring the output of the data_lake
like so: output "s3_logging_bucket { type = required(str) }
. This is then a promise to any consumer of the module outputs that module.data_lake.s3_logging_bucket == null
always will result in false
. It can then be calculated deterministically using static code analysis, and our example count
calculation will know that if it's getting its variable from that output, it's always going to be not null - even though we won't know the actually text value until the resource is deployed to AWS and AWS tells us the actual value.
Related wins:
The added benefit of this approach is that it solves other patterns as well. Currently, for any required module input, you can force the user to provide a value (by just not providing a default one), but you can't enforce that the user doesn't just pass null
instead of an actual value. But in the new world, it would be simple: if you need the input to be non-null, you just specify type = required(str)
and anyone who tries to pass null as an input will get a simple-to-debug error messages without any further headache.
Fwiw, my proposal above is also very similar to the proposal noted above from @mutt13y - except:
coalesce()
downstream), required()
which is the inverse of optional()
- for the symetry, but also because of the strong benefits for static code analysis.Where we have a map or list of map the solution should include the capability for the map key to be absent.
type = list(object({ foo = optional(string,"foo_default") bar = optional(string) }))
so if bar is not provided by the caller the key would be completely absent.
this way you can use the absence of the key to infer some more complex default action rather than having to use a rouge value for default.
Allow me to add a point that seems to be missing from all the proposals above and which also relates to what appears to be a bit of a kludge in how objects are defined today. It's perhaps time to step back and re-look at the object definition per se. If object member fields would follow the same definition syntax as variables, then pretty much no additional syntax rules would be required and it would be neater and more intuitive overall in contrast to the current form of object declaration. It would also likely allow features from the variable declaration, eg validation syntax, to apply to the object fields. Validation on object params today require quite a lot of syntax acrobatics to achieve.
In example form, it would be great to define objects including nullable optional fields like so:
variable "VMCluster" {
type = object({
variable "VMSize" {
type = string
default = "Small"
validation {
condition = can(regex("^Small$|^Medium$|^Large$", var.VMSize))
error_message = "Invalid VM Size."
}
}
variable "OptionalTag" {
type = string
default = null
}
})
}
I would like to suggest a simplified version of @wdec suggestion.
This what we can do today:
variable "my_cluster" {
type = object({
size = string,
tag = string
})
}
The idea is support an alternative syntax that supports the same options as the root variable:
variable "my_cluster" {
type = object({
size = {
type = string
default = "Small"
validation {
condition = can(regex("^Small$|^Medium$|^Large$", var.size))
error_message = "Invalid VM Size."
}
}
tag = {
type = string
default = null
}
})
}
Advantage is that looks familiar as its the same attributes supported by the root variable.
It improves a bit @wdec suggestion by removing the need to write variable
inside which is a bit confusing since the nested type is not really a variable on it's own.
This syntax improves a lot on using any
as it can improve code that uses the lookup
function. It also adds type validation with clear error messages. Another nice thing is to be able to do validation on inner properties.
Thanks.
Same suggestion seems to be described on this issue https://github.com/hashicorp/terraform/issues/24810
By the way, if you sort issues by reactions, this issue comes up at the top over more than one thousand issues:
https://github.com/hashicorp/terraform/issues?q=is%3Aissue+is%3Aopen+sort%3Areactions-%2B1-desc+
So please maintainers, consider it for the next version of Terraform, please!
Same suggestion seems to be described on this issue #24810
This is also roughly what #22449 suggests (supporting defaults for objects with the same semantics as variables, where existence of a default implies optionality)
Lots of cool suggestions. Not sure if that's the right place to mention it, but I also believe that a robust type system should support an either
type or something similar to allow mutually exclusive arguments. Example:
name
- The name of the launch template. If you leave this blank, Terraform will auto-generate a unique name.name_prefix
- Creates a unique name beginning with the specified prefix. Conflicts withname
.
Also:
encrypted
- Enables EBS encryption on the volume (Default: false). Cannot be used withsnapshot_id
.
There's also the case in which for an argument to be specified, another argument should have a specific value. Example:
kms_key_id
- The ARN of the AWS Key Management Service (AWS KMS) customer master key (CMK) to use when creating the encrypted volume.encrypted
must be set totrue
when this is set.
When writting Go code for a Terraform provider it's possible to define a complex schema such as:
"single_header": {
Type: schema.TypeList,
Optional: true,
MaxItems: 1,
Elem: &schema.Resource{
Schema: map[string]*schema.Schema{
"name": {
Type: schema.TypeString,
Required: true,
ValidateFunc: validation.All(
validation.StringLenBetween(1, 40),
// The value is returned in lower case by the API.
// Trying to solve it with StateFunc and/or DiffSuppressFunc resulted in hash problem of the rule field or didn't work.
validation.StringMatch(regexp.MustCompile(`^[a-z0-9-_]+$`), "must contain only lowercase alphanumeric characters, underscores, and hyphens"),
),
},
},
},
},
We just want to have the same on Terraform modules.
Current Terraform Version
Proposal
I like the
object
variable type and it would be nice to be able to define optional arguments which can carrynull
value too, to use:instead of: