Plugin does not support wrapped `nvim` executable?

[tmillr]

I just installed this plugin and followed all the instructions precisely to get it set up and start debugging. As soon as I ran launch() however, I received an error:

E5108: Error executing lua: Vim:Error invoking 'nvim_exec_lua' on channel 4:                                                                                           
Error executing lua: [string "<nvim>"]:1: module 'osv' not found:                                                                                                      
        no field package.preload['osv']                                                                                                                                
        no file '/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/osv.lua'                                                        
        no file '/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/osv/init.lua'                                                   
        no file '/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/osv.so'                                                           
stack traceback:                                                                                                                                                       
        [C]: in function 'require'                                                                                                                                     
        [string "<nvim>"]:1: in main chunk                                                                                                                             
stack traceback:                                                                                                                                                       
        [C]: in function 'rpcrequest'                                                                                                                                  
        ...rt/vimplugin-one-small-step-for-vimkind/lua/osv/init.lua:143: in function 'launch'

I think I know what the issue is, because this is something that I've run into recently with plenary. My nvim and plugins are all managed and installed by the nix pkg manager. To install Neovim, Nix builds a static/readonly wrapper shell script that wraps and exec's the actual nvim binary, and then links this script into PATH as nvim. The reason for the wrapper is to pass several environment vars and cli args automatically directly to nvim, for example to set 'packpath' to the appropriate value to bootstrap plugin loading. Anyway, the crux of the issue comes down to the use of v:progpath which is a readonly variable that is resolved and set internally by nvim. Even when nvim is/points to a wrapper executable/bash script around the actual binary and exec -a "$0" ... is used within the wrapper to invoke it, vim/neovim still sets v:progpath to the actual underlying binary instead of the wrapper script, meaning that none of my plugins get loaded automatically (leading to the error above).

Perhaps it would be possible to allow for the nvim binary spawned by this plugin to be specified/configured (whether via a function arg, or env var, etc.)? This would help out anyone who's using nix or a wrapped nvim binary.

---

On an unrelated note, I was also curious if this plugin allows configuring the communication with the server in order to have it go over a unix domain socket/pipe instead of tcp? The former should be more secure and performant and is also recommended over tcp in the neovim docs. I think nvim actually defaults to using a pipe as well (see :h rpc-connecting).

Thanks

[jbyuki]

Starting from the second question because it's easier to answer. There are two communications channels actually: debuggee <-> DAP Server and DAP server <-> DAP client.

So not be confused, I assume you are talking about the DAP communication channel. It is currently not an option because nvim-dap only has TCP based communication. That's the only reason but it would be technically possible I guess if nvim-dap also agrees to it.

For the first question, you are right, the plugin will directly invoke nvim from v:progpath and disregard any environment variable or option that could have been passed manually or by a script. We had similar issues with LunarVim which also does that. But in that case, the options passed were more predictable and we directly hardcoded them but with Nix it seems more flexible on the launch script side, you might enlighten me as I'm not familiar with this ecosystem at all. Ideally we should just spawn the embedded neovim instance with environments variables and options passed to the current neovim instance I think. I could also make it configurable for the user to pass arguments to the embedded neovim instance. Let me know what you think it's the best for your use case.

[jbyuki]

I added two optional arguments for launch.

• args: A table of arguments passed to the embedded neovim instance in addition to v:progpath, --embed and --headless.
• env: A dictonary containing environment variables in addition to the default ones.

This is subject to change but this way you can test to see if this setup works already.

[tmillr]

It is currently not an option because nvim-dap only has TCP based communication. That's the only reason but it would be technically possible I guess if nvim-dap also agrees to it.

Oh ok thank you for explaining. Maybe I should open an issue over there. Using pipe should work cross-platform and it doesn't have the overhead of creating TCP packets etc. And the nvim docs say that it's somewhat more secure by default. Although I'm wondering if there's a reason for going with TCP/network sockets instead.

I added two optional arguments for launch.

• args: A table of arguments passed to the embedded neovim instance in addition to v:progpath, --embed and --headless.
• env: A dictonary containing environment variables in addition to the default ones.

This is subject to change but this way you can test to see if this setup works already.

Awesome I'll test it out! Thank you

I generally had in mind 3 solutions:

1. Patch vim.progpath on install with the literal path to the nvim wrapper (using either diff patches or sed within Nix)
2. Gain the ability to pass the binary to be invoked by launch() as an argument (with it maybe falling back to v:progpath by default)
3. Translate everything into an ad hoc vimrc (if possible)?

Ideally we should just spawn the embedded neovim instance with environments variables and options passed to the current neovim instance I think.

hmmm well this does give me an additional idea (that I didn't think of before for some reason). In vim you can obviously access env vars, but there's also v:argv that I just found out about recently. So those together with the new launch() options I think I should be able to get it working.

but with Nix it seems more flexible on the launch script side, you might enlighten me as I'm not familiar with this ecosystem at all.

Nix is fairly complex and a bit hard to explain, but it's essentially a package manager and builder that is heavily focused on immutability and reproducibility; all of the packages it ships/distributes are recipes/builds defined as functions so that they are easy to customize via function parameters. Basically when you install or build a pkg, the pkg's function gets called and you have the option to override the func's default parameters. For example for the neovim pkg, one of the parameters is your plugin list. From there Nix builds a more or less immutable nvim that bundles your plugins including anything else configured at build/install time. Part of the output from this built package is a generated, readonly shell script which wraps the underlying/actual nvim binary, and that script glues together all of your configurations and customizations you did at build/install time (i.e. the args you passed to the package function). So yes the shell script, env vars, and cli args can vary quite a bit as the installation/build is highly parametrized and configurable. The proper way to change something is to rebuild as the pkg is supposed to be readonly and immutable (and this is done so that the build parameters correspond precisely to the build output, hence the build is reproducible and the same inputs will always create the same outputs; and this is nice because it means that I can give someone else, or another machine, my build inputs and they will be able to build/reproduce my exact nvim down to a T). All dependencies used by a pkg are strictly versioned and also immutable/readonly and cannot be removed from the fs until every dependent is removed.

You can read more about it here if you'd like. It even has its own Haskell-like language though and has a fairly steep learning curve. I've only been using it for a month or so now, but I think it's pretty cool.

Anyway here's the contents of my generated wrapper script currently:

#! /nix/store/d59v5fpwzza6g95in7qlcwyy6ay1rc9w-bash-5.2-p15/bin/bash -e
export NVIM_SYSTEM_RPLUGIN_MANIFEST='/nix/store/5j13wwkq6bhdwvrm23698jj1y22frhii-neovim-0.9.1/rplugin.vim'
export GEM_HOME='/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/lib/ruby/gems/3.1.0'
PATH=${PATH:+':'$PATH':'}
if [[ $PATH != *':''/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/bin'':'* ]]; then
    PATH=$PATH'/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/bin'
fi
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'';'/';'}
LUA_PATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'';'/';'}
LUA_PATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_CPATH=${LUA_CPATH:+';'$LUA_CPATH';'}
LUA_CPATH=${LUA_CPATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'';'/';'}
LUA_CPATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'$LUA_CPATH
LUA_CPATH=${LUA_CPATH#';'}
LUA_CPATH=${LUA_CPATH%';'}
export LUA_CPATH
export GEM_HOME='/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/lib/ruby/gems/3.1.0'
PATH=${PATH:+':'$PATH':'}
if [[ $PATH != *':''/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/bin'':'* ]]; then
    PATH=$PATH'/nix/store/ivkk793knsacs0m1ddm1g7wnfyi5f7a4-neovim-ruby-env/bin'
fi
PATH=${PATH#':'}
PATH=${PATH%':'}
export PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'';'/';'}
LUA_PATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?/init.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_PATH=${LUA_PATH:+';'$LUA_PATH';'}
LUA_PATH=${LUA_PATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'';'/';'}
LUA_PATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/share/lua/5.1/?.lua'$LUA_PATH
LUA_PATH=${LUA_PATH#';'}
LUA_PATH=${LUA_PATH%';'}
export LUA_PATH
LUA_CPATH=${LUA_CPATH:+';'$LUA_CPATH';'}
LUA_CPATH=${LUA_CPATH/';''/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'';'/';'}
LUA_CPATH='/nix/store/rnjjjd25ms18dmyb47mpnfw17jn5lzss-luajit-2.1.0-2022-10-04-env/lib/lua/5.1/?.so'$LUA_CPATH
LUA_CPATH=${LUA_CPATH#';'}
LUA_CPATH=${LUA_CPATH%';'}
export LUA_CPATH
exec -a "$0" "/nix/store/nlc67haylf28hc306zbhsx9sw7ifnii2-neovim-unwrapped-0.9.1/bin/nvim"  --cmd "lua vim.g.loaded_node_provider=0;vim.g.loaded_python_provider=0;vim.g.python3_host_prog='/nix/store/5j13wwkq6bhdwvrm23698jj1y22frhii-neovim-0.9.1/bin/nvim-python3';vim.g.ruby_host_prog='/nix/store/5j13wwkq6bhdwvrm23698jj1y22frhii-neovim-0.9.1/bin/nvim-ruby'" --cmd "set packpath^=/nix/store/dln53mbfslj7jr225fhqkjnghlmhjli8-vim-pack-dir" --cmd "set rtp^=/nix/store/dln53mbfslj7jr225fhqkjnghlmhjli8-vim-pack-dir" "$@" 

You can see that everything is tightly managed/controlled (the plugin dir, LUA_PATH, PATH, etc.) and all of these /nix/store/* filesystem paths are absolute paths with a hash corresponding to that pkg's inputs (or the inputs/args that the pkg was built/installed with), and are all immutable/readonly as well. It seems that most of the default stuff here mainly has to do with configuring: lua paths, plugin paths (i.e. setting &rtp, &packpath, etc.), and remote plugin hosts (their paths and the paths to their own deps). You can see that ruby is added to PATH because neovim gets its own immutable ruby whose version and deps won't change. This is also necessary because I may or may not even have it available/installed globally (or in my PATH).

Something that's kinda strange is that even though it's doing exec -a $0 which should set arg0 to the wrapper, v:progpath and v:argv[0] still end up as /nix/store/nlc67haylf28hc306zbhsx9sw7ifnii2-neovim-unwrapped-0.9.1/bin/nvim instead. Idk why this is, maybe a platform quirk? (I'm using macOS/darwin)

[jbyuki]

I think I got it now. The arguments ( v:argv ) and the envionment variables ( vim.fn.environ() ) are passed directly to the embedded neovim instance. This works flawlessly now with LunarVim so it should with Nix's launch script as well. I want to say this is a satisfactory solution for now. Let me know if it works.

Thank you for the nice explanation on Nix. I can imagine being very useful for certain needs. I'm familiar with python environments (via anaconda) which is extremely useful so having it for everything makes sense. But anyway, that's a bit off-topic.

I didn't know there was already a request to support named pipes in nvim-dap. Hopefully the request goes through and we can start implementing them in this plugin as well.

[tmillr]

Here's something I tried yesterday and it worked:

require("osv").launch {
    args = vim.tbl_filter(
        (function(i)
            return function(v)
                i = i + 1
                return i ~= 1 and v ~= "--embed" and v ~= "--headless"
            end
        end)(0),
        vim.v.argv
    ),
    port = 8086,
}

(I tried it without passing any env vars just to see what would happen because I think jobstart() inherits env vars from the current nvim process/instance by default?)

Thank you for your help!

[jbyuki]

The environments variables and arguments are actually passed by default now. So you wouldn't need any configuration with the latest commit. But if that worked it should be fine now.

(I tried it without passing any env vars just to see what would happen because I think jobstart() inherits env vars from the current nvim process/instance by default?)

I think in some cases not, but I'm not 100% sure.