Compatibility with Pluto

[dokutan]

Out ouf curiosity I tried running Fennel 1.3.1 on Pluto 0.6.3. Pluto is a fork of Lua 5.4 that claims "99.9% compatibility" with Lua:

• Fennel is not compatible with Pluto, because Pluto adds a few reserved names. It might be useful to add it to the incompatible runtimes in the wiki.
• Simply using an editor to replace new and parent in the source code is enough to make the Fennel script run on Pluto.
• I managed to make most tests succeed using the same renaming approach, note that the pluto binary from the releases page is compiled without support for io.popen, building Pluto from source removes that issue.
• When running Fennel on Pluto a few warnings about excessive arguments are produced. I am unable to decide whether these findings should be considered bugs in Fennel.

./pluto fennel-1.3.1
fennel-1.3.1:2053: warning: too many arguments [excessive-arguments]
    2053 | provided = safe_compiler_env(false)
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here: expected 0 arguments, got 1.
fennel-1.3.1:2287: warning: too many arguments [excessive-arguments]
    2287 | return add_macros(macro_loaded[modname], ast, scope, parent_)
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here: expected 3 arguments, got 4.
fennel-1.3.1:2414: warning: too many arguments [excessive-arguments]
    2414 | return add_macros(macro_tbl, ast, scope, parent_)
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here: expected 3 arguments, got 4.
fennel-1.3.1:4119: warning: too many arguments [excessive-arguments]
    4119 | parse_string(b)
         | ^^^^^^^^^^^^^^^ here: expected 0 arguments, got 1.
fennel-1.3.1:4987: warning: too many arguments [excessive-arguments]
    4987 | local _135_0, _136_0 = f0(k, x)
         | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here: expected 1 argument, got 2.
Welcome to Fennel 1.3.1 on PUC Lua 5.4!
Use ,help to see available commands.
Try installing readline via luarocks for a better repl experience.
>> 

[technomancy]

This is interesting, but I don't think it's a good idea to rename all these locals just for compatibility with a fork.

When running Fennel on Pluto a few warnings about excessive arguments are produced. I am unable to decide whether these findings should be considered bugs in Fennel.

They're definitely not bugs, just unnecessary junk that can be safely deleted. Well, except for the last one:

fennel-1.3.1:4987: warning: too many arguments [excessive-arguments] 4987 | local _135_0, _136_0 = f0(k, x) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ here: expected 1 argument, got 2.

This one is actually a bug in Pluto. Here's the relevant function:

(fn kvmap [t f ?out]
  "Map function f over key/value table t, similar to above, but it can return a
sequential table if f returns a single value or a k/v table if f returns two.
Optionally takes a target table to insert the mapped values into."
  (let [out (or ?out [])
        f (if (= (type f) :function)
              f ; <- unknown number of arguments
              #(. $ f))] ; <- one argument
    (each [k x (stablepairs t)]
      (match (f k x) ; <- pluto complains here
        (key value) (tset out key value)
        (value) (table.insert out value)))
    out))

Note that f here is either a hashfn of one argument or an unknown function passed in by the caller. Pluto seems to assume that it is always the former, but the second argument is needed in the latter case, and removing it as advised would cause a failure.

[jaawerth]

Since the fennel compiler is self-hosted, if you wanted to use it "live" (as it should work fine doing AOT compilation), one could actually use fennel to create a custom build that doesn't use those symbol names.

I'm not sure if we can consider this a supported part of the API, since internal variable names are by definition an implementation detail, but right now Fennel will actually alias the "manged" (turning a fennel identifier, which is much more permissive, into one that works with lua) form of local declarations if they have already been declared up-scope.

With that in mind, all one needs to do in order to compile fennel without the reserved names is use a parent scope where they already exist. Because I was curious whether it would work as well as I thought, I wrote up a quick example and hooked it into some basic CLI code to make a PoC command line tool you can run like the following to create a custom build.

# custom fennel.lua
RESERVED_WORDS='switch continue enum new class parent export' \
  ./compile-reserved.fnl path/to/fennel/src/fennel.fnl \
  pluto-friendly-fennel.lua

# custom fennel launcher
RESERVED_WORDS='switch continue enum new class parent export' \
  ./compile-reserved.fnl path/to/fennel/src/launcher.fnl \
  pluto-friendly-fennel

@technomancy Since the anti-shadowing logic in this isn't an official part of the API, if there are other lua variants that also work except for certain reserved words, it might not hurt to have a compiler option for this just so people can build without relying on aspects of behavior that might change. I don't have a strong opinion on this, though.

compile-reserved.fnl CLI

The relevant bits are the compile-file and reserved-scope functions; the rest are mostly generic for writing a CLI tool with file io.

#!/usr/bin/env fennel
(local {: sym : list &as fennel} (require :fennel))

(local dir-sep (package.config:sub 1 1))
(local path-sep (package.config:sub 3 3))

(local default-opts {:requireAsInclude true :compiler-env _G})

(local help "Usage: ./compile-reserved.fnl path/to/src/fennel.fnl [outfile.lua]\n
ENVIRONMENT VARIABLES
* RESERVED_WORDS - space-delimited list of words to treat as reserved symbols\n")

(λ dirname [path]
  (select -1 (path:find (table.concat ["(.+)%" "+[^" "]*[" "]-$"] dir-sep))))

(λ copy-to [tgt ...]
  (faccumulate [t tgt i 1 (select :# ...)]
    (collect [k v (pairs (pick-values 1 (select i ...))) &into t]
      k v)))

(λ reserved-scope [reserved-words ?compiler-opts]
  (let [scope (fennel.scope (?. ?compiler-opts :scope))
        bind-reserved (icollect [_ v (ipairs reserved-words) &into (list)]
                        (sym v))]
   ;; Builds `(local (every reserved word) nil)` expression, which we will then
   ;; compile, throwing away the output. this ensures the parent of the child
   ;; scope we return contains the reserved words
   (fennel.compile (list (sym :local) bind-reserved (sym :nil))
                   (copy-to {} (or ?compiler-opts {}) {: scope}))
   (fennel.scope scope)))

(λ compile-file [infile ?reserved-words]
  "Compile infile to lua, aliasing any reserved words. Returns lua string."
  (let [add-fennel-path (.. (or (dirname infile) "") dir-sep "?.fnl")
        opts (copy-to {} default-opts {:filename infile})]
    (set fennel.path (.. add-fennel-path path-sep fennel.path))

    (when (and ?reserved-words (< 0 (length ?reserved-words)))
      (set opts.scope (reserved-scope ?reserved-words opts)))

    (with-open [fin (io.open infile)]
      (fennel.compile-string (fin:read :*a) opts))))

(λ write-file [filename data]
  (if (= :- filename)
    (io.stdout:write (.. data "\n"))
    (with-open [fout (assert (io.open filename :w))]
      (fout:write (.. data "\n")))))

(let [reserved (case (os.getenv :RESERVED_WORDS)
                 wordlist (icollect [word (wordlist:gmatch "%S+")]
                            word))]
  ;; Parse command-line arguments
  (case arg
    [infile outfile] (write-file outfile (compile-file infile reserved))
    [infile] (write-file :- (compile-file infile reserved))
    (where (or :-h :--help)) (io.stdout:write help)
    _ (do (io.stderr:write help) (os.exit 1))))

[technomancy]

I don't want to change the built-in list of reserved words without having some specific criteria for what constitutes a good reason to do so in future cases, (otherwise it sets an unsustainable precedent) but I would take a patch which allowed additional words to be added in the options table.

[jaawerth]

Agreed - changing the built-in would require us to keep up with various other nonstandard lua distributions or variants, but I think it's a reasonable change to allow an API-supported way to add them that could very well get rid of the only thing blocking usage on those platforms.

[technomancy]

Going to close this out but if anyone wants to take a shot at supplying a config option for reserved words that would be cool.

[technomancy]

This is fixed by #477; thanks!