Question about deps.sh and embed.sh

[katawful]

I am looking into ways to unify my macros across various uses, between my configs and multiple plugins, so I don't have to make sure I have to keep them all up to date manually. For my Aniseed plugins, I was looking at embed.sh, which says that it puts the result of deps.sh into the lua folder of the Aniseed enabled project. However, when I go to require the macros from the embedded repo, they can't be found. I have tried various pathnames for the macro call, but none seem to work

The repo I am testing this feature out on is found here: https://github.com/katawful/test-fennel

[Olical]

Hmm did you try test-fennel.katcros-fnl.macros.nvim.api.utils.macros? Since it's embedded under your Lua directory which is placed in the path. I worked out that path by looking in the Lua dir and just walking the directories down to your macros file.

I know that's a little weird but the tradeoffs between inventing new path ideas and just piggiebacking on the Lua path are difficult. I went with "just put it under Lua and use that path system as a base" which works well but leads to this duplication of .fnl files into the lua dir which smells off but doesn't cause much harm for the value it provides (when it works...).

[katawful]

Yeah, that doesn't seem to work still for whatever reason

[katawful]

I think the issue is that because these are macros, the fennel compiler needs them in the fennel path. Unlike runtime files I embed via embed.sh, which just get the output lua files in the appropriate lua folder of my project. Should there be a script to embed a project within fnl/?

[Olical]

Just looking into this now because Conjure embeds Aniseed's macros inside of it's own Lua directory and they can be referenced fine.

I think you're right with the path issue, so you could manage that yourself by requiring aniseed.fennel and calling the add-path function in there if your plugin isn't on your path for some reason.

What I think is happening here (could be wrong!) is that you're working on your plugin without it being added to your path, so just as a directory of Fennel files. This technically works for evals but it doesn't include the directory in the path like Aniseed expects.

So when I'm working on an Aniseed based plugin I'll always either include a https://github.com/Olical/nvim-local-fennel file that adds my CWD to the runtimepath at startup OR I'll do what I do 99% of the time: Point my dotfiles plugin manager at the directory I'm working with like: "~/repos/Olical/conjure" {:mod :conjure}

This means the plugin and it's Lua directory is added to the path and is ready for evals. This is a caveat I should document somewhere in Aniseed I guess 🤔 that if you're working on a plugin you need it in your Neovim path or Fennel / Lua paths won't find your modules.

[Olical]

If you are doing this already then something else is afoot of course. It could be that your lua or runtimepath's aren't being set in the way that I expect. Aniseed does this sync at startup so your Fennel path reflects all of the equivalent Neovim paths. This, I think, is the least surprising way for this to work. If Lua plugins can see it, so can Fennel etc.

(defn sync-rtp [compiler]
  "Synchronises the runtime paths into the fennel.macro-path"
  (let [;; For direct macros under the fnl dir.
        fnl-suffix (.. fs.path-sep "fnl" fs.path-sep "?.fnl")

        ;; For macros embedded from other tools under your lua dir.
        lua-suffix (.. fs.path-sep "lua" fs.path-sep "?.fnl")

        rtps (nvim.list_runtime_paths)
        fnl-paths (a.map #(.. $ fnl-suffix) rtps)
        lua-paths (a.map #(.. $ lua-suffix) rtps)]
    (tset compiler :macro-path (str.join ";" (a.concat fnl-paths lua-paths)))))

[katawful]

Fennel/lua paths isn't something I'm super familiar with. I tried looking around in the fennel table, but not sure what exactly I should be seeing or doing within the module

[Olical]

So if you can make sure your plugin is loaded by your plugin manager, even though you're still working on it, the paths will be managed for you automatically, it doesn't require you to modify any paths since plugin managers / neovim put your plugin's lua dir on the path and Aniseed automatically picks that up and syncs it with Fennel for you.

One alternative is to require aniseed.fennel (may be prefixed like conjure.aniseed.fennel or whatever you plugin's prefix is) and then call (fnl.add-path "path-to-my-project/fnl/?.fnl").

Another approach could be to do something like the Aniseed internal compiler script

#!/usr/bin/env bash

# Compiles all Fennel code into Lua assuming you have Aniseed cloned through dep.sh.
# Usage: deps/aniseed/scripts/compile.sh

nvim --headless -u NONE \
    -c "let &runtimepath = &runtimepath . ',deps/aniseed,' . getcwd()" \
    -c "lua package.path = package.path .. ';$(pwd)/lua/?.lua;deps/aniseed/lua/?.lua'" \
    -c "lua require('aniseed.compile').glob('**/*.fnl', 'fnl', 'lua')" \
    +q

I'm modifying the runtimepath to add my current directory which Aniseed will pick up automatically and sync into Fennel's path.

[katawful]

So I think I sorta got the hang of it, but it's not working how I expect still. Added my test repo to my Neovim package manager (packer if it matters), then Conjure started working perfectly. However, Aniseed still wouldn't compile with my macros (path being: plugin-name.macro-embed-name.macro-path). When I copied the Aniseed compile script, and added the path for the sourced macro repo added from dep.sh, it would then compile with macro-embed-name.macro-path, but not plugin-name.macro-embed-name.macro-path. This change in path now means Conjure won't work

Test plugin for seeing what I did: https://github.com/katawful/test-fennel

[katawful]

Ok, figured out the Aniseed compiling issue. The makefile made by seed.sh puts compile.sh before any embed.sh. Swapping those two scripts lets me properly compile. I think this is an issue with Aniseed then?

[Olical]

Ohh yeah it may well be! I can't remember why it's in that order, maybe there's a reason for it so it won't be as simple as swapping (since it'll break something else, maybe?) but I'll have a look. If swapping works for you, please use that for now, I'll see if it's all the same for me and swap it in the repo too if it seems okay.

[Olical]

I think I've fixed this on the develop branch! So the embed is before the compile now and compile supports a little sed magic to ensure your compiled lua refers to your embedded Aniseed copy. Maximum flexibility and you can properly depend on macro files of your deps now. I hope this helps!

[katawful]

New method seems to work well in my test repo, thanks for the help!