Parsing seems not to work in LiveBook, but works in IEX

[chgeuer]

Installation into Livebook

Mix.install([
  {:hxl, github: "drowzy/hxl", app: false}
])

* Getting hxl (https://github.com/drowzy/hxl.git)
remote: Enumerating objects: 753, done.        
remote: Counting objects: 100% (753/753), done.        
remote: Compressing objects: 100% (464/464), done.        
remote: Total 753 (delta 347), reused 635 (delta 235), pack-reused 0        
origin/HEAD set to master
Resolving Hex dependencies...
Dependency resolution completed:
New:
  nimble_parsec 1.1.0
* Getting nimble_parsec (Hex package)
==> nimble_parsec
Compiling 4 files (.ex)
Generated nimble_parsec app
==> hxl
Compiling 1 file (.yrl)
src/hcl_parser.yrl: Warning: conflicts: 54 shift/reduce, 0 reduce/reduce
Compiling 1 file (.erl)
Compiling 22 files (.ex)
Generated hxl app

:ok

Seems not to generate any statements

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.decode_as_ast()

{:ok, %HXL.Ast.Body{statements: []}}

The (missing) whitespace seems to trip things off?

"""
resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}
"""
|> HXL.decode_as_ast()

{:error, "syntax error before: 1:37 "}

[drowzy]

I tried the following:

elixir -e 'Mix.install([{:hxl, github: "drowzy/hxl", app: false}])
    """

    resource "upcloud_server" "server1" {
      plan = "1xCPU-1GB"
    }

    """
    |> HXL.decode_as_ast() |> inspect(pretty: true) |> IO.puts()'

which seems to work for me.

expected output:

{:ok,
 %HXL.Ast.Body{
   statements: [
     %HXL.Ast.Block{
       body: %HXL.Ast.Body{
         statements: [
           %HXL.Ast.Attr{
             expr: %HXL.Ast.TemplateExpr{delimiter: nil, lines: ["1xCPU-1GB"]},
             name: "plan"
           }
         ]
       },
       labels: ["upcloud_server", "server1"],
       type: "resource"
     }
   ]
 }}

Could you verify if the above works for you? And i'll try to reproduce using livebook.

[drowzy]

I did the following:

Mix.install([{:hxl, github: "drowzy/hxl", app: false}])
"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.decode_as_ast()

Also if possible could you attach the output of:

"""
resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}
""" |> HXL.Lexer.tokenize()

Thanks!

[chgeuer]

It's strange... Within LiveBook it doesn't work, in IEX all is well...

I start LiveBook like this

docker run -p 8080:8080 --pull always -u "$(id -u):$(id -g)" "livebook/livebook"

install

Mix.install([{:hxl, github: "drowzy/hxl", app: false}])

output

* Getting hxl (https://github.com/drowzy/hxl.git)
remote: Enumerating objects: 753, done.        
remote: Counting objects: 100% (753/753), done.        
remote: Compressing objects: 100% (464/464), done.        
remote: Total 753 (delta 347), reused 635 (delta 235), pack-reused 0        
origin/HEAD set to master
Resolving Hex dependencies...
Dependency resolution completed:
New:
  nimble_parsec 1.1.0
* Getting nimble_parsec (Hex package)
==> nimble_parsec
Compiling 4 files (.ex)
Generated nimble_parsec app
==> hxl
Compiling 1 file (.yrl)
src/hcl_parser.yrl: Warning: conflicts: 54 shift/reduce, 0 reduce/reduce
Compiling 1 file (.erl)
Compiling 22 files (.ex)
Generated hxl app

:ok

HXL.decode_as_ast()

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.decode_as_ast()

{:ok, %HXL.Ast.Body{statements: []}}

HXL.decode_as_ast()

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.decode_as_ast()

{:ok, %HXL.Ast.Body{statements: []}}

[drowzy]

Strange indeed! I followed your setup of livebook and did it in two blocks i.e Mix.install in one and parsing in one but still couldn't reproduce it. Does anything else work? Try:

HXL.decode_as_ast("a = 1")

Also would you mind attaching the output of:

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

""" |> HXL.Lexer.tokenize()

and

"""
resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}
"""
|> HXL.Lexer.tokenize()

Thank you!

[drowzy]

I committed a lexer fix to ignore carriage returns + tabs. Could you try it out and let me know if it solves the issue? Thank you!

[chgeuer]

Thanks Simon, that fixed it, great!

Mix.install([{:hxl, github: "drowzy/hxl", app: false}])

* Getting hxl (https://github.com/drowzy/hxl.git)
remote: Enumerating objects: 771, done.        
remote: Counting objects: 100% (771/771), done.        
remote: Compressing objects: 100% (476/476), done.        
remote: Total 771 (delta 357), reused 648 (delta 240), pack-reused 0        
origin/HEAD set to master
Resolving Hex dependencies...
Dependency resolution completed:
New:
  nimble_parsec 1.1.0
* Getting nimble_parsec (Hex package)
==> nimble_parsec
Compiling 4 files (.ex)
Generated nimble_parsec app
==> hxl
Compiling 1 file (.yrl)
src/hcl_parser.yrl: Warning: conflicts: 54 shift/reduce, 0 reduce/reduce
Compiling 1 file (.erl)
Compiling 22 files (.ex)
Generated hxl app

:ok

---

HXL.decode_as_ast("a = 1")

{:ok,
 %HXL.Ast.Body{statements: [%HXL.Ast.Attr{expr: %HXL.Ast.Literal{value: {:int, 1}}, name: "a"}]}}

---

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.Lexer.tokenize()

{:ok,
 [
   {:identifier, {2, 8}, ["resource"]},
   {:string, {2, 25}, ["upcloud_server"]},
   {:string, {2, 35}, ["server1"]},
   {:"{", {2, 37}},
   {:identifier, {3, 6}, ["plan"]},
   {:=, {3, 8}},
   {:string, {3, 20}, ["1xCPU-1GB"]},
   {:"}", {4, 1}}
 ], "", %{}, {6, 68}, 68}

---

"""
resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}
"""
|> HXL.Lexer.tokenize()

{:ok,
 [
   {:identifier, {1, 8}, ["resource"]},
   {:string, {1, 25}, ["upcloud_server"]},
   {:string, {1, 35}, ["server1"]},
   {:"{", {1, 37}},
   {:identifier, {2, 6}, ["plan"]},
   {:=, {2, 8}},
   {:string, {2, 20}, ["1xCPU-1GB"]},
   {:"}", {3, 1}}
 ], "", %{}, {4, 64}, 64}

---

"""

resource "upcloud_server" "server1" {
  plan = "1xCPU-1GB"
}

"""
|> HXL.decode_as_ast()

{:ok,
 %HXL.Ast.Body{
   statements: [
     %HXL.Ast.Block{
       body: %HXL.Ast.Body{
         statements: [
           %HXL.Ast.Attr{
             expr: %HXL.Ast.TemplateExpr{delimiter: nil, lines: ["1xCPU-1GB"]},
             name: "plan"
           }
         ]
       },
       labels: ["upcloud_server", "server1"],
       type: "resource"
     }
   ]
 }}