paultyng
commented
4 years ago This is potentially related to (or superseded by support for) the pseudo dynamic type, #248.
Open radeksimko opened 4 years ago
paultyng
commented
4 years ago This is potentially related to (or superseded by support for) the pseudo dynamic type, #248.
radeksimko
commented
4 years ago @paultyng do you mind explaining the relationship a bit more? 🤔
I'm imagining that regardless of how the data is represented in state or config, we need to address the above main problems (validation & diffing) for each language (JSON, YAML, XML, ...) separately.
I suppose it can help with data representation in state, but we'd then still need to write some parsers which parse raw JSON/YAML/XML into internal cty's dynamic type, because I expect users will want to keep raw format in their configs, so the "dynamic" type really ends up being just an internal representation.
It might help users in accessing data inside the structure from CRUD as we'd pre-parse it for them I guess? I don't understand how that's related to this issue though.
paultyng
commented
4 years ago Yeah, I think it depends on how folks want to use those types of data. For example, instead of supporting a JSON or YAML attribute, one could do something like:
attribute = jsondecode(<<JSON
{
"foo": "bar",
"baz": 3
}
JSON
# or
attribute = yamldecode(<<YAML
foo: bar
baz: 3
YAML
# being equivalent to
attribute = {
foo = "bar"
baz = 3
}And then diffing (and diff rendering), etc could all be handled via the dynamic type. If providers went in this direction for their support, they wouldn't need type hinting as it wouldn't matter, they would just set it as dynamic.
paultyng
commented
4 years ago This is of course about a subset of the type hinting you mention above, dates, base64, etc or other complex strings we'd want to retain as strings, still could use some type extensions on top of string.
radeksimko
commented
4 years ago @paultyng I see! That's a good point.
I reckon that users would still want to see it represented as the original format (JSON/YAML,...) in the plan output and everywhere else. It would be confusing if we just displayed HCL-style diffs for these fields.
i.e. from user's perspective this should be just a hidden implementation detail.
Problem Statement (current situation)
Providers can today specify primitive types, such as
TypeStringorTypeIntto fields which store more complex types. Different providers take different approaches to store complex types, but these are common situations:TypeString)TypeString)TypeString)TypeIntor RFC3339/ISO8601 timestamp asTypeString)TypeString)A few problems arise with such "dummy" representation across all providers as described below.
Validation
While providers (provider developers) are aware that certain fields are XML/YAML/JSON and should never contain arbitrary string, it is often not trivial or not obvious how to validate that. We have some prior art in that area:
validation.ValidateJsonStringvalidation.ValidateRFC3339TimeStringbut no consistent story.
As a result providers often rely on server-side API validation and end up making unnecessary round trips with invalid data that could have been caught at plan-time, or implement their own validation and effectively duplicate work that was already done & tested in other provider(s).
Diffing
Terraform 0.12 applies simple heuristics to render JSON differences in a human-readable way, but this is currently only implemented for JSON.
As a result differences of such complex-typed fields such as YAML are not human readable as these are just rendered as single line of text, e.g.
Additionally most (all?) providers want to suppress no-op diffs, such as whitespace changes. They use
DiffSuppressFuncwith custom implementations orstructure.SuppressJsonDiffto achieve this. Again - we have some prior art but no consistent story here.State
To ensure consistency and correct diff calculation provider needs to ensure these values are saved in their "canonical" form to the state, which is often achieved via
StateFuncand we have some prior art in that area:structure.NormalizeJsonStringbut no consistent story.
Customisability
While many providers could be satisfied with simple "JSON" or "YAML" field, there are providers which likely require custom rules for diffing. Diffing of AWS IAM policies is one great example - AWS IAM API doesn't document any canonical format of policies and often treats things like
"values": ["single"]and"values": "single"as equal and interchangeable.Proposal
This will likely require full RFC and some discussion/scrutiny, but rough plan is below:
command/formatmore easily customisable/pluggableJust for the sake of better search-ability here are some names other people have used when referring to this problem/feature: virtual types, diff hints, dynamic attribute diffing
Related: https://github.com/hashicorp/terraform/issues/21817
Related for core - function parity for working with these types and converting.