Supporting heredoc

[munjalpatel]

Hey @drowzy,

I am trying to parse a config with heredoc like this:

action "transform_data" {
  js = <<-EOT
    const output = { ...input, x: 1 };
    return output;
  EOT
}

But it is failing with the following error:

** (FunctionClauseError) no function clause matching in HXL.Error.upcase_first/1

    The following arguments were given to HXL.Error.upcase_first/1:

        # 1
        'syntax error before: '

    Attempted function clauses (showing 1 out of 1):

        defp upcase_first(<<char::utf8, rest::binary>>)

    (hxl 0.1.2) lib/hxl/error.ex:37: HXL.Error.upcase_first/1
    (hxl 0.1.2) lib/hxl/error.ex:24: HXL.Error.format_reason/1
    (hxl 0.1.2) lib/hxl/parser.ex:21: HXL.Parser.parse/1
    (hxl 0.1.2) lib/hxl.ex:95: HXL.decode/2
    (hxl 0.1.2) lib/hxl.ex:25: HXL.decode_file/2
    iex:3: (file)

Is parsing HCL that contains heredoc supported? If not, are there any potential workarounds?

BTW, this is a great library, excellent work!

[munjalpatel]

Found out that it is supported. https://github.com/drowzy/hxl/blob/2f9295c10c8d3d4a2f6090683e9aa62d5d903705/src/hcl_parser.yrl#L196

So, fixed my code by adding a missing space like below.

action "transform_data" {
  js = <<- EOT
    const output = { ...input, x: 1 };
    return output;
  EOT
}

However, it seems that the library is trying to parse string inside heredoc as well since now I am getting this error:

{:error, "Unrecognized input: ';\n    return output;\n  EOT\n}\n' at 19:238"}

[drowzy]

Thanks for the issue! You are correct that it should treat the lines in the heredoc as text and not parse it. Seems to be a bug. I’ll look into it!

[munjalpatel]

Thanks Simon :)

[drowzy]

I tried with the following:

hcl = """
action "transform_data" {
  js = <<-EOT
    const output = { ...input, x: 1 };
    return output;
  EOT
}
"""

iex(8)> HXL.Lexer.tokenize(hcl)
{:ok,
 [
   {:identifier, {1, 6}, ["action"]},
   {:string_part, {1, 23}, ["transform_data"]},
   {:"{", {1, 25}},
   {:identifier, {2, 4}, ["js"]},
   {:=, {2, 6}},
   {:heredoc, {0, 0}},
   {:identifier, {2, 13}, ["EOT"]},
   {:text, {4, 0}, ["    const output = { ...input, x: 1 };"]},
   {:text, {5, 0}, ["    return output;"]},
   {:identifier, {5, 5}, ["EOT"]},
   {:"}", {6, 1}}
 ], "", %{}, {7, 106}, 106}

and

HXL.Parser.parse(hcl)

{:ok,
 %HXL.Ast.Body{
   statements: [
     %HXL.Ast.Block{
       type: "action",
       labels: ["transform_data"],
       body: %HXL.Ast.Body{
         statements: [
           %HXL.Ast.Attr{
             name: "js",
             expr: %HXL.Ast.TemplateExpr{
               delimiter: "EOT",
               lines: ["    const output = { ...input, x: 1 };",
                "    return output;"]
             }
           }
         ]
       }
     }
   ]
 }}

Which seems to be correct. Can you post a complete example? Might be something else that is wrong. Or atleast the output from HXL.Lexer.tokenize/1.

[drowzy]

Oh and both <<-EOT and <<- EOT is allowed now, in the second example that you posted it skipped tokenize as a heredoc due to the space, which it should do now :).

[munjalpatel]

The only difference is that you used tokenized while I am using devode_file

test.hcl

action "transform_data" {
  js = <<- EOT
    const output = { ...input, x: 1 };
    return output;
  EOT
}

hcl_parser.ex

defmodule HclParser do
  def parse do
    HXL.decode_file("test.hcl")
  end
end