Unexpected Token parsing HCL/Terraform files with conditional statements.

[dvpierce]

Error:

  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 126, in feed_token
    action, arg = states[state][token.type]
                  ~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: '__ANON_3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/hcl2/api.py", line 14, in load
    return loads(file.read(), with_meta=with_meta)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/hcl2/api.py", line 27, in loads
    tree = hcl2.parse(text + "\n")
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/lark.py", line 652, in parse
    return self.parser.parse(text, start=start, on_error=on_error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parser_frontends.py", line 101, in parse
    return self.parser.parse(stream, chosen_start, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 41, in parse
    return self.parser.parse(lexer, start)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 171, in parse
    return self.parse_from_state(parser_state)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 193, in parse_from_state
    raise e
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 184, in parse_from_state
    state.feed_token(token)
  File "/home/chrobinson.com/pierdavpa/.local/lib/python3.11/site-packages/lark/parsers/lalr_parser.py", line 129, in feed_token
    raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
lark.exceptions.UnexpectedToken: Unexpected token Token('__ANON_3', 'secondary_read_write_connection_string_id') at line 70, column 5.
Expected one of: 
        * STAR
        * COMMA
        * PERCENT
        * MINUS
        * __ANON_1
        * SLASH
        * __ANON_8
        * RBRACE
        * __ANON_5
        * __ANON_2
        * __ANON_4
        * __ANON_9
        * __ANON_7
        * LESSTHAN
        * PLUS
        * MORETHAN
        * __ANON_6
        * QMARK
        * __ANON_0

When parsing a file that has a conditional assignment in it. key = value == null ? "" : someothervalue

Any idea how I might fix or work around this? Or if a library fix is needed (or can even address this?)

[purajit]

I have this issue too, it seems like it's expecting parentheses.

[kkozik-amplify]

Hello, thanks for the report I'm not able to reproduce. Does the bug still occur? If so, could you provide some piece of config that's causing it?

[grahamhar]

Hi @kkozik-amplify ,

I just hit the same and have managed to update the test case to show the failure (see this branch). I have been unable to find a solution so far as I am not that familiar with lark.

[kkozik-amplify]

@grahamhar Hi, thanks for the test case. It looks like the issue is caused by incorrect definition in the grammar file - we will look into this. As a workaround, wrapping conditional expression into parentheses should help: operator = (local.assertion == 1 ? local.yes : local.no)

[grahamhar]

It does appear that the files parse OK when we downgrade to 4.3.0

I have been doing a little more testing and we have some conditionals that have embedded conditionals too.

operator = local.assertion == 1 ? local.some_logic == 2 ? local.yes : "custom" : local.no

[Awlie]

Hi there, we also had this issue. We currently downgraded to 4.3.0. Any higher version (4.3.1, 4.3.2, 4.3.3) didn't work

We could only reproduce this error this way in our unit test:

module "unittest" {
  nested = {
    nested_attribute = var.nested == null ? null : {
      publisher = var.nested.publisher
    }
    nested_after_conditional_object = var.variable
  }
}

and (basically the same)

module "unittest" {
  nested = {
    nested_attribute = var.attr == null ? local.yes : local.no
    nested_after_conditional_object = var.variable
  }
}

which would fail with lark.exceptions.UnexpectedToken: Unexpected token Token('__ANON_3', 'nested_after_conditional_object')

other conditionals wouldn't cause issues.

The workaround with parentheses also worked, but is not a viable option for us

These didn't fail: almost the same as above, but without the attribute after the conditional (nested_after_conditional_object)

module "unittest" {
  nested = {
    nested_attribute = var.nested == null ? null : {
      publisher = var.nested.publisher
    }
  }
}

some other conditionals we tested (we also tested the failing one with parentheses, which passes)

module "unittest" {
  conditional        = var.question ? one(azurerm_resource_group.rg[*].name) : var.no
  nested_conditional = var.question != null ? var.yes != null ? var.yesyes : local.yesno : var.no
  attribute = var.attribute == null ? null : {
    publisher = var.attribute.publisher
  }
  after_attribute = var.variable
}

(this module syntax is of course not correct, but it doesn't matter for the parser) This is pretty much the test-case that grahamhar added.

[ariker]

Here is another file where this happens: https://github.com/Azure/terraform-azurerm-naming/blob/0.4.1/main.tf

[kadigergo]

here is also an example:

locals {
  a = {
    b = terraform.workspace == "production" ? "x" : "y"
    c = terraform.workspace == "production" ? "n" : "m"
  }
}

parsing fails only if it is nested and we have two with the ? : syntax

[RPitt]

This seemingly is related to / the same root cause as https://github.com/amplify-education/python-hcl2/issues/163 https://github.com/amplify-education/python-hcl2/issues/140 In short parsing of expressions seems to be problematic in 4.3.1+