New Parser: cloudformation template.yaml

georgettica commented 1 month ago

Hello!

I wanted to use cat template.yaml | jc --yaml for processing, but the yaml formatter didn't know how to cope with the schema

an example yaml snipper

# yaml-language-server: $schema=https://github.com/aws/serverless-application-model/raw/develop/samtranslator/validator/sam_schema/schema.json
# also read https://github.com/redhat-developer/vscode-yaml/issues/669 in order to make most of the warning disappear
Parameters:
  age:
    Type: Number
    Default: 10
Conditions:
  IsOld: !Equals [!Ref age, 10]

the error I get is:

$ cat template.yaml | LC_ALL=C jc -d --yaml
Traceback (most recent call last):
  File "/opt/homebrew/bin/jc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 965, in main
    JcCli().run()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 947, in run
    self._run()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 911, in _run
    self.standard_parse_and_print()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 803, in standard_parse_and_print
    self.create_normal_output()
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/cli.py", line 755, in create_normal_output
    self.data_out = self.parser_module.parse(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/jc/parsers/yaml.py", line 158, in parse
    for document in yaml.load_all(data):
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/main.py", line 476, in load_all
    yield constructor.get_data()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 110, in get_data
    return self.construct_document(self.composer.get_node())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 125, in construct_document
    for _dummy in generator:
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 633, in construct_yaml_map
    value = self.construct_mapping(node)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 429, in construct_mapping
    return BaseConstructor.construct_mapping(self, node, deep=deep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 244, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
    data = self.construct_non_recursive_object(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 181, in construct_non_recursive_object
    data = constructor(self, node)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/jc/1.25.3/libexec/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 647, in construct_undefined
    raise ConstructorError(
ruamel.yaml.constructor.ConstructorError: could not determine a constructor for the tag '!Equals'
  in "<unicode string>", line 6, column 10

I searched online and found:

https://github.com/gristlabs/yaml-cfn | in js but parses the cloudformation as a yaml and has more stars
https://github.com/puresec/aws-parsecf | in python (has a parse_yaml function)

if you want me to help let me know. if I were to do it I could add a jc --cloudformation and parse the yaml with the special parser or write one myself

the comments in the beginning of the template are there for further reading and how other tools partially implement supporting cloudformation

kellyjonbrazil commented 1 month ago

Cool - yeah I think that sounds like a good new parser!

georgettica commented 3 weeks ago

as seen in this issue I linked to here, I found a very nice and documented repo, that makes the code work perfectely.

I don't want to just copy it and maintain myself / push the code here and I am seeing if they are ok with extracting the code to a generic section which will be in python so I can work with it in the PR.

let me know if that seems like a good idea / if you have a different approach to this / what I wrote is not reflected in the issue I created and I will correct and reconsolidate what I wrote to fit into what I want to happen

kellyjonbrazil commented 3 weeks ago

Could work, but not sure. Since the project is coded in Golang I think you'd still need to install the Go library/dependencies before using the Python bindings, so it couldn't simply be "vendored" in like we have done with other dependencies. (see the x.509 parsers) The license seems pretty open, so that's good. This seems more like a custom parser route because even if we included a compiled version of the command and used subprocess, this would be architecture-dependent.

georgettica commented 3 weeks ago

That is great insight and I didn't even think this was an option to pull in the binary.

If the license permits, I'll try to incorporate the golang code to python (like I tried before you said but now that the license is good I can also feel better just taking this code snippet and pushing here)

I am concerned that the internal yaml implementation they have is way more stable and easy to extend than in python.

kellyjonbrazil / jc

New Parser: cloudformation template.yaml #580