Using `folke/lazy.nvim` with module-name/string plugin definitions

[datwaft]

Hello!

I am currently having this issue (https://github.com/folke/lazy.nvim/issues/104) when trying to use hotpot.nvim with lazy.nvim.

Do you have any idea of what could be done to work around that issue?

I am opening this as an issue in case someone also has problems when trying to use hotpot.nvim with lazy.nvim.

My current idea would be to first load hotpot.nvim and then load lazy.nvim, but I don't know how that could be done while at the same time letting lazy.nvim keep hotpot.nvim updated.

Maintainer Edit

Jump to solutions:

• Prebuild into lua: https://github.com/rktjmp/hotpot.nvim/issues/97#issuecomment-1363945602
• Preload into table: https://github.com/rktjmp/hotpot.nvim/issues/97#issuecomment-1364240883

[rktjmp]

Yeah I figured this might come up. I have not used Lazy yet. AFAIK it does (or did) replace every part of neovim plugin infra, inside and outside of lua.

Possibly you can force hotpots lua loader in front of Lazy's but then you're probably discarding any reason to use Lazy if you're intercepting its loader.

I wont commit to supporting Hotpot x Lazy at this time, not for any philosophical reason, I just don't have the time to get in the guts of it to see how to patch them together.

FWIW I have been kicking around the idea of an option to not compile to cache, which might make it lazy compatible, where hotpot only does the "find fnl/ make lua/" in its loader and then returns nil to kick off to the regular finder. AFAIK the byte-code-cache that impatient and hotpot (and lazy) use is intended to be upstreamed into neovim core at some point and non-cache compiles are required for compatibility with that.

I can't say whether that would fix it with Lazy though as I haven't looked at the exact failure state.

[rktjmp]

https://github.com/rktjmp/hotpot.nvim/tree/colocation

Test this in a container first!!!

• You must enable the colocate = true option in setup, you should probably also nuke any existing ~/.cache/nvim/hotpot dir.
• Once enabled, hotpot will build any fnl/ files into lua/, and then "gives up" and expects other lua infra to take over.
• The bytecode cache is disabled, so something like impatient (or lazy?) is required to re-enable that kind of functionality.
• ~/.config/nvim/lua must exist at load for nvim to find files in lua/
  • This might be a bug in nvim, if you make ~/.config/nvim/fnl/test.fnl, boot nvim, :lua require"test", the file is created in ~/.config/nvim/lua/test.lua but the require fails. Restarting nvim fixes it, or creating ~/.config/nvim/lua/ before the first run also fixes it.
  • Probably if the dir doesnt exist at startup nvim does not add it to the rtp.

This is experimental (!!!) for now, option names may change, etc etc etc. Quite likely it will be mainlined though as it increases hotpots general ... intergrability.

You'll have to do some leg work yourself to integrate both into the loader chain, at least for now.

All I can tell you is that hotpot inserts itself first in package.loaders after it's required (see hotpot.fnl), and I imagine lazy does this too, so you need to find some spot somewhere that you can load hotpot, grab its loader, load lazy then reinsert hotpots loader into position 1 (it must be first so it can find the fennel code obviously).

Probably

local hotpot = require("hotpot")
local hotpot_loader = package.loaders[1]
local lazy = require("lazy")
lazy.setup({...})
table.insert(package.loaders, 1, hotpot_loader) -- cant say if this is too late or not, possibly lazy needs a "pre-loaders" option to insert before its own depending on what it does *after* that insertion.

In the future I could imagine exposing the hotpot loader a bit more ergonomically and you might be able to do something like

local hotpot = require("hotpot")
local lazy = require("lazy")
lazy.setup({...})
table.insert(package.loaders, 1, hotpot.loader) -- field does not exist yet, right now

or probably

local hotpot = require("hotpot")
local lazy = require("lazy")
lazy.setup({...})
hotpot.reinsert_loader() -- inserts at 1, ensures no duplicates (eg: hp-loader, lazy, hp-loader (from setup), ...)

Very appreciative of any testing you can do with this idea.

[datwaft]

Good news!

Your branch is working!

Here is when I open nvim for the first time:

Here is when I open nvim the second time:

You'll have to do some leg work yourself to integrate both into the loader chain, at least for now.

The problem with that is that hotpot cannot be required before lazy is setup, it says that the module doesn't exist. 😢

[rktjmp]

Wait, even before you call setup? Lazy steals the loader before it's setup and stops you from requiring anything? Does it redefine _G.require? Perhaps it could save it somewhere like _G.rawrequire before it overwrites it?

Hotpot ships with a .lua file that should load just as any other lua file - and it then internally bootstraps the hotpot/fnl code (this might be funky with colocation enabled...).

[datwaft]

My guess is that Hotpot is not in the runtimepath until Lazy is loaded. Maybe we can force it to be on the runtimepath by guessing where Hotpot is installed and adding that to the runtimepath.

[rktjmp]

Ah right cause you're installing via Lazy, yes it does not put anything in the RTP from what I know, or it replaces it with its own or something.

Yeah I would just force where ever lazy installs hotpot to into the RTP, see if that makes things work, then if so perhaps folke would be nice enough to expose some (lazy.where-is :package-name) kind of interface to let you get the always-correct rtp path or something.

Hotpot's bootstrapper should be resilient enough to bootstrap into the correct path (eg hotpot.nvim/lua/hotpot should have more than just fennel.lua in it after loading).

[folke]

Just make sure you add hotpot to the RTP before requiring hotpot. That should definitely work.

[folke]

lazy adds a loader at 2, to make sure any pre-loaders still work. Lazy also byte-compiles everything, so that should be fine.

so there's no need to replace the hotpot loader. Just do:

vim.opt.rtp:append("path to installed hotpot")
pcall(require, "hotpot") -- would fail if not yet installed. Restart Neovim after initial install to load your fnl files
local lazy = require("lazy")
lazy.setup({...})

[folke]

To add to my previous comments, plugins do get added to the rtp as usual when they get loaded. But of course in this case the plugin was not loaded yet

[folke]

Thinking more about this, hotspot should just work without that new colocate option. But only if you pass a table spec to lazy. Passing a spec module would not work.

[rktjmp]

This is how I would solve it via hotpot.api.make.

https://github.com/rktjmp/hotpot.nvim-x-lazy.nvim

There is a small :chicken: :egg: moment where you must run nvim, to install lazy and hotpot, hotpot builds the plugin configs immediately but lazy doesn't see notice the additions to the lua/plugins dir, so you have to restart nvim and re-run lazy.

As configured, hotpot only compile changed files on boot but you can setup a uv watcher (I think there is an example of that in the docs or cookbook, or fwatch.nvim for example) to rebuild the files on change, or a command, or binding to rebuild current file or whatever.

It seems that Hotpot in general works fine with Lazy, just the specific collision around defining plugins. They both provide the byte-cache though and hotpots loader precedence probably interferes with all of Lazy's smarts.

I never really liked having the custom cache -- I'd rather leverage existing tools -- so probably the colocation option will still make it into main and when enabled it'll just use lazy/impatient. It's not possible to use no-index + no-colocation though as other loaders wont be able to find the files and non-colocation was kind of a core idea of hotpot so the options kind of at odds with the future (or past) perhaps.

To add to my previous comments, plugins do get added to the rtp

My fault, I meant "installed into the (default) rtp".

[datwaft]

I just tried this and it doesn't seem to work:

-- Bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({
        "git",
        "clone",
        "--filter=blob:none",
        "--single-branch",
        "https://github.com/folke/lazy.nvim.git",
        lazypath,
    })
end

-- Add lazy.nvim to rtp
vim.opt.runtimepath:prepend(lazypath)
-- Add hotpot.nvim to rtp
local hotpotpath = vim.fn.stdpath("data") .. "/lazy/hotpot.nvim"
vim.opt.runtimepath:prepend(hotpotpath)

if pcall(require, "hotpot") then
    -- Configure hotpot.nvim
    require("hotpot").setup({
        provide_require_fennel = true,
        enable_hotpot_diagnostics = false,
    })
    -- Compile plugin configurations
    -- This is required because Lazy requires in some way those plugins and
    -- hotpot doesn't detect that they have been required
    for file in vim.fs.dir(vim.fn.stdpath("config") .. "/fnl/conf/plugins") do
        file = file:match("^(.*)%.fnl$")
        require("conf.plugins." .. file)
    end
end

-- Configure lazy.nvim
require("lazy").setup("conf.plugins", {
    performance = {
        cache = {
            enabled = false,
        },
    },
})

The idea was to load Hotpot before Lazy was loaded by adding it to rtp, and it works, it loads Hotpot first, but Lazy doesn't detect the compiled fnl/conf/plugins modules.

I will now try it with hotpot.api.make.

[folke]

As I said before, you can't use a plugins module. Change it to the below so hotpot can do its thing

[folke]

-- Bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({
        "git",
        "clone",
        "--filter=blob:none",
        "--single-branch",
        "https://github.com/folke/lazy.nvim.git",
        lazypath,
    })
end

-- Add lazy.nvim to rtp
vim.opt.runtimepath:prepend(lazypath)
-- Add hotpot.nvim to rtp
local hotpotpath = vim.fn.stdpath("data") .. "/lazy/hotpot.nvim"
vim.opt.runtimepath:prepend(hotpotpath)

local plugins = {"rktjmp/hotpot.nvim"}
if pcall(require, "hotpot") then

    -- Configure hotpot.nvim
    require("hotpot").setup({
        provide_require_fennel = true,
        enable_hotpot_diagnostics = false,
    })
    -- Compile plugin configurations
    -- This is required because Lazy requires in some way those plugins and
    -- hotpot doesn't detect that they have been required
       plugins = {}
    for file in vim.fs.dir(vim.fn.stdpath("config") .. "/fnl/conf/plugins") do
        file = file:match("^(.*)%.fnl$")
        plugins[#plugins+1] = require("conf.plugins." .. file)
    end
end

-- Configure lazy.nvim
require("lazy").setup(plugins, {
    performance = {
        cache = {
            enabled = false,
        },
    },
})

[folke]

I think something like that should work.

If hotpot is not available it will be installed the first time. After that plugins will be a list of the requires of all your plugin files.

Then lazy setup should load all your real plugins.

[folke]

performance wise, not using a plugins module is not a big deal here, since hotpot also caches and byte-compiles your files.

Regular lua files would still be cached by lazy (unless hotspot also compiles those?)

[folke]

you can probably still enabled the lazy cache in that code too

[datwaft]

As I said before, you can't use a plugins module. Change it to the below so hotpot can do its thing:

require("lazy").setup(require("conf.plugins"), ...)

Yes, don't worry, just testing every possibility.

I will test that one in a bit.

---

About using hotpot.api.make, I just tested it and it seems to work:

-- Bootstrap lazy.nvim
local lazypath = vim.fn.stdpath("data") .. "/lazy/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({
        "git",
        "clone",
        "--filter=blob:none",
        "--single-branch",
        "https://github.com/folke/lazy.nvim.git",
        lazypath,
    })
end

-- Add lazy.nvim to rtp
vim.opt.runtimepath:prepend(lazypath)
-- Add hotpot.nvim to rtp
local hotpotpath = vim.fn.stdpath("data") .. "/lazy/hotpot.nvim"
vim.opt.runtimepath:prepend(hotpotpath)

if pcall(require, "hotpot") then
    -- Configure hotpot.nvim
    local hotpot = require("hotpot")
    hotpot.setup({
        provide_require_fennel = true,
        enable_hotpot_diagnostics = false,
    })
    -- Compile plugin configurations
    local conf_path = vim.fn.stdpath("config") .. "/fnl/conf/plugins"
    hotpot.api.make.build(
        conf_path,
        { verbosity = 0, atomic = false },
        -- ~/user/.config/nvim/fnl/plugins/.../x.fnl
        -- ^ root ^^^^^^^^^^^^     ^ path ^^^^^^^^^^
        "(.*)/fnl/(.*)",
        function(root, path, util)
            return util["join-path"](root, "lua", path)
        end
    )
end

-- Configure lazy.nvim
require("lazy").setup("conf.plugins", {
    performance = {
        cache = {
            enabled = false,
        },
    },
})

It is not an ideal solution because it pollutes the lua/ directory.

Edit: changed the typo from fnl/ to lua/ 😅

---

I think something like that should work.

I like that solution, I will test it in a bit.

[datwaft]

I think something like that should work.

Just tested that solution and it works!!!

I added a more complex example with nvim-treesitter and an init.fnl to verify that it worked too with a more complex configuration.

See https://github.com/datwaft/lazy-hotpot_issue/tree/works-1 for how I implemented it.

Another advantage of that workaround is that I don't need to create a fnl/conf/plugins folder just for hotpot, as it is added as the first element of the table.

[folke]

Great! :) You should try enabling the cache as well. Should still work and will be good for caching lua plugins

[datwaft]

you can probably still enabled the lazy cache in that code too You should try enabling the cache as well. Should still work and will be good for caching lua plugins

I also tested this and it seems to work, so I will have it enabled when I try to migrate to Lazy, which probably will be later today or tomorrow.

[rktjmp]

Regular lua files would still be cached by lazy (unless hotspot also compiles those?)

It actually does, as it normally returns the regular lua loader for lua files (otherwise we'd do all the searching to find fnl or lua files, compare, then just make the next loader do all the searching again, so we just do both sides "anyway").

Since it returns the loader in both cases, it also byte-caches both (well, technically only the lua but from either source...) - otherwise stuff like impatient would do half and hotpot would do half, bit of a mess.

Leaving Lazy's cacher on should be fine. At worst its a no-op, at best it might cache something hotpot doesn't touch.

It is not an ideal solution because it pollutes the fnl/ directory.

Do you mean lua/ directory? The make function as given shouldn't be putting anything in fnl/.

I believe going the non-make-api direction wont get you Lazy's config-changed watchers as there's no files to watch, so there's some give and take to which solution you use, no artefacts :sparkles: or watchable artefacts :eyes:. If you're ok restarting nvim on changes anyway though ...

[datwaft]

Do you mean lua/ directory?

Yes, sorry. 😅

If you're ok restarting nvim on changes anyway though ...

Actually I prefer that, I usually save the file like 5 times every second (hyperbole) when I am editing something so with Lazy the popup that something has changed appears every time.

[datwaft]

In case someone comes upon this issue while trying to make Hotpot and Lazy work together, I just finished my migration to Lazy so you can use my dotfiles as a reference:

datwaft/nvim.conf#93fcc61

[bb010g]

Is any special configuration needed to properly use Hotpot alongside lazy.nvim now that both lazy.nvim (https://github.com/folke/lazy.nvim/commit/6b55862d2d264f0b48e0b9e42cc2d14f136bed55, https://github.com/folke/lazy.nvim/blob/dac844ed617dda4f9ec85eb88e9629ad2add5e05/lua/lazy/init.lua#L36-L39) and Hotpot (#101, #107, https://github.com/rktjmp/hotpot.nvim/blob/cfe493572fd80678855c3fd3c4cba9f2fec840de/fnl/hotpot/loader/init.fnl) use vim.loader?

[datwaft]

I am using it right now and I am not having any problem.

[rktjmp]

The install instructions use lazy as a demo, so please open an issue or PR if those are incorrect!