Elixir package support does not work on Windows

[Smaehtin]

The recent addition of support for Elixir packages in Gleam 0.23 does not work on Windows. Attempting to compile a project that depends on tzdata gives the following:

PS C:\Users\dev\projects\test> gleam build
  Resolving versions
Downloading packages
 Downloaded 11 packages in 0.11s
  Compiling certifi
===> Analyzing applications...
===> Compiling certifi
  Compiling parse_trans
===> Analyzing applications...
===> Compiling parse_trans
  Compiling unicode_util_compat
===> Analyzing applications...
===> Compiling unicode_util_compat
  Compiling ssl_verify_fun
error: Program not found

The program `elixir` was not found. Is it installed?

The issue, I believe, is that the invoking Elixir is currently hardcoded to invoking elixir, while on Windows it should be done via elixir.bat. However, fixing this in the compiler gives the following error:

  Compiling ssl_verify_fun
DEBUG writing_elixir_paths_to_build
DEBUG command_exec program="elixir.bat" args="--eval File.write!(\"gleam_elixir_paths\", [:eex, :elixir, :logger, :mix] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(\"\\n\"))" env=[("TERM", "dumb")] cwd=Some("build\\dev\\erlang")
** (SyntaxError) nofile:1:38: unexpected token: ]
    |
  1 | File.writeeex, :elixir, :logger, :mix] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(\\n\))
    |                                      ^
    (elixir 1.13.3) lib/code.ex:403: Code.validated_eval_string/3
DEBUG reading_file path="build\\dev\\erlang\\gleam_elixir_paths"
DEBUG removing_failed_build package=ssl_verify_fun
DEBUG deleting_directory path="build\\dev\\erlang\\ssl_verify_fun"
DEBUG directory_did_not_exist_for_deletion path="build\\dev\\erlang\\ssl_verify_fun"
ERROR Failed error=FileIo { kind: File, action: Read, path: "build\\dev\\erlang\\gleam_elixir_paths", err: Some("The system cannot find the file specified. (os error 2)") }
error: File IO failure

An error occurred while trying to read this file:

    build\dev\erlang\gleam_elixir_paths

My guess is it has something to do with the way command-line arguments are passed to elixir.bat that has to be done differently on Windows than on Unix-like operating systems.

[lpil]

Thank you.

fyi @tynanbe

[lpil]

@Smaehtin could you share what you have so far for fixing it? The .bat bit sounds very useful 🙏

[ninjaferret]

This stackoverflow post suggests that you use double double-quotes, i.e. "", rather than trying to escape with \", for windows command-line stuff, e.g.

...args="--eval File.write!(""gleam_elixir_paths"",...

[tynanbe]

A Windows alternative for the line in question might look like this, since the single \ characters escape for Rust.

"File.write!(\"\"{}\"\", [{}] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(\"\"\\n\"\"))"

To end up as

"File.write!(""gleam_elixir_paths"", [:eex, :elixir, :logger, :mix] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(""\n""))"

[Smaehtin]

Thanks for the suggestions! Unfortunately, it seems like a combination of the ! and : characters is still giving problems after correctly escaping the other characters.

I have boiled the problem down to the following batch script (test.bat) by looking at what's going on inside elixir.bat:

@echo off
setlocal enabledelayedexpansion

set "VAR=%~1"
echo "%VAR%"

test.bat "File.write!("test.txt", [:eex])" gives the following output (Not valid code, just using it to show the problem): "File.writeeex])" For some reason, the argument gets "cut off", and I have no idea how to correctly escape the input.

It can be boiled down even further by test.bat "Hello!wo:rld" which outputs: "Hellorld"

I can't seem to find a way to correctly escape !, the output doesn't change if I use ^! or ^^!.

Unfortunately, I have basically zero experience with batch scripts, so I'm a bit clueless here.

[lpil]

That is super useful, thank you @Smaehtin !

[tynanbe]

@Smaehtin thanks for testing. @lpil 's suggestion is to replace File.write!( with :ok = File.write(, as the final(?) piece of the puzzle.

[Smaehtin]

@Smaehtin thanks for testing. @lpil 's suggestion is to replace File.write!( with :ok = File.write(, as the final(?) piece of the puzzle.

Yes! Great workaround. That fixed the ! issue. Now it seems like there's a new problem with the way Rust passes (well, doesn't) quotes to the shell.

DEBUG command_exec program="elixir.bat" args="--eval :ok = File.write(\"gleam_elixir_paths\", [:eex, :elixir, :logger, :mix] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(\"\\n\"))" env=[("TERM", "dumb")] cwd=Some("build\\dev\\erlang")   
** (SyntaxError) nofile:1:18: unexpected token: "\" (column 18, code point U+005C)
    |
  1 | :ok = File.write(\gleam_elixir_paths\, [:eex, :elixir, :logger, :mix] |> Stream.map(fn(lib) -> lib |> :code.lib_dir |> Path.expand end) |> Enum.join(\\n\))
    |                  ^
    (elixir 1.13.3) lib/code.ex:403: Code.validated_eval_string/3

The quotes get stripped from the input. Seems like https://github.com/rust-lang/rust/issues/29494 is related and it requires using raw_arg from std::os::windows::process::CommandExt.

Will do some more digging tomorrow if I have time

[lpil]

Another hack, but we could use an s sigil instead of a quoted string if we can't figure out the escaping.

:ok = File.write(~s(gleam_elixir_paths), ...

Does anyone have a WIP branch for this? A PR would be fab!

[Smaehtin]

Another hack, but we could use an s sigil instead of a quoted string if we can't figure out the escaping.

:ok = File.write(~s(gleam_elixir_paths), ...

Does anyone have a WIP branch for this? A PR would be fab!

Perfect! That avoids a lot of headache. I'll have a PR ready soon.

[sophiebits]

So arguably this seems like a bug in elixir.bat, except that it seems like batch scripts just cannot correctly handle arbitrary characters in their arguments: https://stackoverflow.com/q/4200316 – so that seems unlikely to be properly fixed.

Perhaps a more reliable fix that doesn't rely on rewriting the code would be to use a temp file instead of --eval.