Dynamic blocks

[traut]

Background

The content structure in FCL templates is static at the moment. Fabric supports content filtering during rendering (#99), but users can't mutate the document structure based on data.

Requirements

• be able to create multiple documents from data defined with data block
• be able to create multiple content and section blocks from data defined with data block in the template
• support jq query for producing a list to iterate over for multiple items

Design

We introduce dynamic block type. It is somewhat similar to Terraform's dynamic blocks but has a different syntax.

dynamic block works with specific block types, similar to config block: dynamic block supports document, section and content blocks.

For example, dynamic document would look like this:

dynamic document "test_doc" {

  # Normal data definition. This data will be used for dynamic `for_each_query` query, so
  # this means that all data definitions must be executed _before_ `for_each_query` query is executed

  data inline "docs" {
    doc_texts = [
      { title = "A", text = "a" },
      { title = "B", text = "b" }
    ]
  }

  # Dynamic block attributes
  for_each_query = ".data.inline.docs.doc_texts"

  title = "Document {{ .dynamic.item.title }}"

  content text {
    value = "Doc text is {{ .dynamic.iteam.text }} with index {{ .dynamic.item_index }}"
  }
}

[!NOTE] Dynamic blocks extend the blocks they wrap. For example dynamic document block is document block with additional attributes, so data blocks can be defined. In dynamic content block no nested blocks (apart from meta) can be defined.

dynamic block attributes:

• for_each_query -- (optional) a string value, a JQ query. The result of the query provides a collection to iterate over.
• for_each_item_name -- (optional) a string attribute, a name for the current element of a collection in the context. By default, it is item
• condition_query -- (optional) a string attribute, a JQ query. The result of the query is converted to bool.

Dynamic blocks produce the block they wrap:

• for every element in the collection returned by for_each_query
• when condition_query returns a response that we treat as true: boolean true or non-null object

For example, this snippet shows how we can choose which block to render based on available data:

dynamic content text {
  query = ".data.inline.my_elements"
  condition_query = "(.query_result == {} or .query_result == [] or .query_result == null) | not"

  value = "The elements: {{ .data.inline.my_elements }}"
}

dynamic content text {
  query = ".data.inline.my_elements"
  condition_query = ".query_result == {} or .query_result == [] or .query_result == null"

  value = "There are no elements"
}

Behaviour

Execution flow

• if it is a dynamic document block, all data blocks are executed and set in the context
• if it is a dynamic content block and query is defined, query is executed and .query_result value set in the context
• if condition_query is defined, it is executed.
  • if the query returns a boolean false or null, the dynamic block is skipped completely
  • if the query returns a boolean true or a non-null object, the execution continues
• if for_each_query is defined, it is executed and the blocks that match the number of items returned by for_each_query are created

Context for produced blocks

Dynamic block extends the context available for produced blocks with these values:

• .dynamic.item or .dynamic.<item-name> -- the current element of the collection
• .dynamic.item_index -- the index of the current element of the collection

If the dynamic block has another dynamic block nested inside (if section is wrapped in dynamic block), .dynamic.* values in the context are overwritten by the values for the inner block.

Naming

If the dynamic block is named, the names of produced blocks are suffixed with .dynamic.item_index value, for example section.foo[<.dynamic.item_index>].

If the block is unnamed, it's given a random name and the names of the copies are suffixed too: for example, section.24a086f[<.dynamic.item_index>].

Example

Dynamic section and dynamic content block:

document "test_doc" {

  data inline "docs" {
    doc_texts = [
      { title = "A", text = "a" },
      { title = "B", text = "b" }
    ]
  }

  title = "Parent doc with {{ len .data.inline.docs.doc_texts }} docs"

  dynamic section {
    for_each_result_in_query = ".data.inline.docs.doc_texts"

    title = "Section about document {{ .dynamic.item.title }}"

    content text {
      value = "Doc text is {{ .dynamic.item.text }} with index {{ .dynamic.item_index }}"
    }

    dynamic content text {
      for_each_query = "[\"foo\", \"bar\"]"

      query = ".data.inline.docs.doc_texts[.dynamic.item_index].title"

      value = "Dynamic text: item={{ .dynamic.item }}, index={{ .dynamic.item_index }}, query_result={{ .query_result }}"
    }
  }
}

renders into

# Parent doc with 2 docs

## Section about document A

Doc text is a with index 0

Dynamic text: item=foo, index=0, query_result=A

Dynamic text: item=bar, index=1, query_result=B

## Section about document B

Doc text is b with index 1

Dynamic text: item=foo, index=0, query_result=A

Dynamic text: item=bar, index=1, query_result=B

The ref blocks should still be supported:

dynamic section ref {
  for_each_result_in_query = ".data.inline.docs.doc_texts"

  base = section.foo
}

and https://github.com/blackstork-io/fabric/issues/29 is supported as well.

Dynamic documents

The behaviour of the dynamic document blocks is slightly more complex.

dynamic document "test_doc" {

  # Normal data definition. This data will be used for dynamic `for_each_query` query, so
  # this means that all data definitions must be executed _before_ `for_each_query` query is executed

  data inline "docs" {
    doc_texts = [
      { title = "A", text = "a" },
      { title = "B", text = "b" }
    ]
  }

  # Dynamic block attributes
  for_each_query = ".data.inline.docs.doc_texts"
  for_each_item_name = "my_item"

  title = "Document {{ .dynamic.my_item.title }}"

  content text {
    value = "Doc text is {{ .dynamic.my_iteam.text }} with index {{ .dynamic.item_index }}"
  }
}

renders into 2 separate documents:

# Document A

Doc text is a with index 0

and

# Document B

Doc text is b with index 1

Data blocks

for_each_query relies on data blocks to be already executed and available in .context.

Since data blocks do not use the context, it should be possible to execute them only once and pass the results in the produced document block contexts.

CLI issues

• render CLI command should support dynamic.document.<document-name> as a target
• data CLI command should support dynamic.document.<document-name>.data.<...> as a target

The behaviour of publish / output flags in CLI should be adjusted in dynamic blocks as target introduce ambiguity.

References

• https://github.com/hashicorp/terraform/issues/21219
• https://github.com/blackstork-io/fabric/issues/70
• https://github.com/blackstork-io/fabric/issues/126
• https://github.com/hashicorp/hcl2/blob/master/ext/dynblock/
• https://developer.hashicorp.com/terraform/language/expressions/dynamic-blocks

[dobarx]

I think dedicated dynamic be a bit easier to implement and understand than embedding these parameters and functionality in selected few section & document blocks.

Not using dedicated block for this would move all content blocks to the lower level on content tree. There is no way to use this dynamic feature to create blocks on the top level of the document or on the same level where it is used.

[traut]

@dobarx I almost wrote the whole issue about dynamic blocks! I decided against it because I kept inventing the behavior that reminded me very much of the combo of section / document blocks!

The problem with a dedicated dynamic block is that we must define its behavior in all situations: Where can it be placed? How does it behave when defined inside and outside the document? If it is outside, where does the data come from? Should we allow data blocks inside dynamic block? Should we reference blocks outside dynamic block on the root level? If we allow data blocks inside dynamic blocks, it opens the door to non-document-level data blocks, which will create another set of problems. Can dynamic blocks be referenced? etc

If we take a step back, dynamic block is just a container around some content with a couple of instructions. And we already have 2 containers with defined behaviour -- document and section blocks! Extending these containers allows us to build up on all behaviours the blocks support, which is so much easier.

Not using dedicated block for this would move all content blocks to the lower level on content tree. There is no way to use this dynamic feature to create blocks on the top level of the document or on the same level where it is used.

It might need a dedicated unpacking step, indeed, where all these "dynamic" blocks are unpacked into "normal" blocks. We don't need to retain the knowledge that some nodes were originally dynamic—we can adjust the template tree and continue.

[dobarx]

Should we allow data blocks inside dynamic block? Should we reference blocks outside dynamic block on the root level? If we allow data blocks inside dynamic blocks, it opens the door to non-document-level data blocks, which will create another set of problems.

But doesn't allowing document being dynamic open the door for data blocks being also dynamic already? That inline data in given example is already dynamic. My understanding of dynamic is that we should take for_each property of it and take all children blocks of it and recreate them dynamically without executing them. Allowing any block to be dynamic.

[traut]

But doesn't allowing document being dynamic open the door for data blocks being also dynamic already?

they are dynamic in a sense that they will be copied as part of the document, but they still follow the rules and limitations we established for them

in the description above, for_each controls the behavior of the block it is defined in (not the children), so if it is inside document, multiple document blocks will be created (the same for section). This allows us to use the existing definitions without defining new ones: for example, if for_each would affect only children, we would need to invent a block that can include document to have dynamic documents.

Potentially, we can allow for_each for every block type -- since it affects the block it is defined in, it can be applied to any block. It might be interesting to do in the future but for the first iteration, enabling for_each only for document and section is easier to explain and to use.

[traut]

I've updated the issue description with the new design