amplify-education / python-hcl2

MIT License
253 stars 56 forks source link

UnexpectedToken: Unexpected token Token('FOR', 'for') #166

Closed zhcli closed 2 months ago

zhcli commented 3 months ago

version 4.3.4

example tf file - https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_alert_prometheus_rule_group

resource "azurerm_monitor_alert_prometheus_rule_group" "example" {
  name                = "example-amprg"
  location            = "West Europe"
  resource_group_name = azurerm_resource_group.example.name
  cluster_name        = azurerm_kubernetes_cluster.example.name
  description         = "This is the description of the following rule group"
  rule_group_enabled  = false
  interval            = "PT1M"
  scopes              = [azurerm_monitor_workspace.example.id]
  rule {
    enabled    = false
    expression = <<EOF
histogram_quantile(0.99, sum(rate(jobs_duration_seconds_bucket{service="billing-processing"}[5m])) by (job_type))
EOF
    record     = "job_type:billing_jobs_duration_seconds:99p5m"
    labels = {
      team = "prod"
    }
  }

  rule {
    alert      = "Billing_Processing_Very_Slow"
    enabled    = true
    expression = <<EOF
histogram_quantile(0.99, sum(rate(jobs_duration_seconds_bucket{service="billing-processing"}[5m])) by (job_type))
EOF
    for        = "PT5M"
    severity   = 2

    action {
      action_group_id = azurerm_monitor_action_group.example.id
    }

    alert_resolution {
      auto_resolved   = true
      time_to_resolve = "PT10M"
    }

    annotations = {
      annotationName = "annotationValue"
    }

    labels = {
      team = "prod"
    }
  }
  tags = {
    key = "value"
  }
}

example python code

import hcl2
with open('test.tf', 'r') as file:
    dict = hcl2.load(file)

exception

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py:77](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py#line=76), in ParserState.feed_token(self, token, is_end)
     76 try:
---> 77     action, arg = states[state][token.type]
     78 except KeyError:

KeyError: 'FOR'

During handling of the above exception, another exception occurred:

UnexpectedToken                           Traceback (most recent call last)
Cell In[2], line 2
      1 with open('test.tf', 'r') as file:
----> 2     dict = hcl2.load(file)

File [~/python-hcl2-venv/lib/python3.12/site-packages/hcl2/api.py:14](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/hcl2/api.py#line=13), in load(file, with_meta)
      8 def load(file: TextIO, with_meta=False) -> dict:
      9     """Load a HCL2 file.
     10     :param file: File with hcl2 to be loaded as a dict.
     11     :param with_meta: If set to true then adds `__start_line__` and `__end_line__`
     12     parameters to the output dict. Default to false.
     13     """
---> 14     return loads(file.read(), with_meta=with_meta)

File [~/python-hcl2-venv/lib/python3.12/site-packages/hcl2/api.py:27](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/hcl2/api.py#line=26), in loads(text, with_meta)
     18 """Load HCL2 from a string.
     19 :param text: Text with hcl2 to be loaded as a dict.
     20 :param with_meta: If set to true then adds `__start_line__` and `__end_line__`
     21 parameters to the output dict. Default to false.
     22 """
     23 # append new line as a workaround for https://github.com/lark-parser/lark/issues/237
     24 # Lark doesn't support a EOF token so our grammar can't look for "new line or end of file"
     25 # This means that all blocks must end in a new line even if the file ends
     26 # Append a new line as a temporary fix
---> 27 tree = hcl2.parse(text + "\n")
     28 return DictTransformer(with_meta=with_meta).transform(tree)

File ~/python-hcl2-venv/lib/python3.12/site-packages/lark/lark.py:658, in Lark.parse(self, text, start, on_error)
    640 def parse(self, text: str, start: Optional[str]=None, on_error: 'Optional[Callable[[UnexpectedInput], bool]]'=None) -> 'ParseTree':
    641     """Parse the given text, according to the options provided.
    642 
    643     Parameters:
   (...)
    656 
    657     """
--> 658     return self.parser.parse(text, start=start, on_error=on_error)

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parser_frontends.py:104](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parser_frontends.py#line=103), in ParsingFrontend.parse(self, text, start, on_error)
    102 kw = {} if on_error is None else {'on_error': on_error}
    103 stream = self._make_lexer_thread(text)
--> 104 return self.parser.parse(stream, chosen_start, **kw)

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py:42](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py#line=41), in LALR_Parser.parse(self, lexer, start, on_error)
     40 def parse(self, lexer, start, on_error=None):
     41     try:
---> 42         return self.parser.parse(lexer, start)
     43     except UnexpectedInput as e:
     44         if on_error is None:

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py:88](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py#line=87), in _Parser.parse(self, lexer, start, value_stack, state_stack, start_interactive)
     86 if start_interactive:
     87     return InteractiveParser(self, parser_state, parser_state.lexer)
---> 88 return self.parse_from_state(parser_state)

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py:111](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py#line=110), in _Parser.parse_from_state(self, state, last_token)
    109     except NameError:
    110         pass
--> 111     raise e
    112 except Exception as e:
    113     if self.debug:

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py:102](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser.py#line=101), in _Parser.parse_from_state(self, state, last_token)
    100 for token in state.lexer.lex(state):
    101     assert token is not None
--> 102     state.feed_token(token)
    104 end_token = Token.new_borrow_pos('$END', '', token) if token else Token('$END', '', 0, 1, 1)
    105 return state.feed_token(end_token, True)

File [~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py:80](http://localhost:8888/lab/tree/~/python-hcl2-venv/lib/python3.12/site-packages/lark/parsers/lalr_parser_state.py#line=79), in ParserState.feed_token(self, token, is_end)
     78 except KeyError:
     79     expected = {s for s in states[state].keys() if s.isupper()}
---> 80     raise UnexpectedToken(token, expected, state=self, interactive_parser=None)
     82 assert arg != end_state
     84 if action is Shift:
     85     # shift once and return

UnexpectedToken: Unexpected token Token('FOR', 'for') at line 27, column 5.
Expected one of: 
    * __ANON_3
    * RBRACE
zhcli commented 2 months ago

@kkozik-amplify thanks for reviewing/merging the PR. Since v4.3.4 there are a number improvements, may I know when the next release would be?

kkozik-amplify commented 2 months ago

@zhcli released v4.3.5. It also allows if and for_each as identifiers. Thanks for contributing!