feat: make all plugins opt by default

[lewis6991]

• Renamed spec.opt to spec.lazy
• Added spec.start
• Make sure to load plugin dependencies before loading the plugin
• Allow nested dependency specs
• Only allow a plugin to be specified as a table once. This is tracked with plugin.simple.

[lewis6991]

@wbthomason

[lewis6991]

The only issue with making all plugins opt is that it forces the user to specify plugin order for dependencies.

E.g. with start mode I could have:

      'hrsh7th/cmp-nvim-lsp',
      'hrsh7th/cmp-nvim-lsp-signature-help',
      'hrsh7th/cmp-buffer',
      'hrsh7th/cmp-emoji',
      'hrsh7th/nvim-cmp',

and this would work fine since all plugins are added to rtp at the same time (effectively).

However, with opt mode, you can't do this and need something like:

  {'hrsh7th/nvim-cmp',
    requires = {
      'hrsh7th/cmp-nvim-lsp',
      'hrsh7th/cmp-nvim-lsp-signature-help',
      'hrsh7th/cmp-buffer',
      'hrsh7th/cmp-emoji',
  }
}

However, this doesn't even work since cmp and its sources depend on each other. I guess this it what wants and after try to solve.

So I'm still trying to think of a good solution for this. I spoke with folke about this and I think the right thing to do might be.

1. Add all plugins to rtp (but not source plugin files). This is sort of what start does.
2. Source plugin dependencies (and their configs)
3. Source plugin
4. Load plugin config

So, I think start plugins do have value, and I think we should support a start mode. opt mode will be similar to opt_default on master except the plugins are loaded automatically.

I think I'll tweak this PR so start mode is default, but for development I'm going to use opt mode.

[lewis6991]

I've got changes on top of this that'll be tricky to conflict resolve so merging this and will make said changes later.

[clason]

@lewis6991 could you explain a bit what the current state is after this PR got merged? I must admit I'm still confused after reading the description and comments here. Are plugins now opt by default? What is the rationale for that change? If I prefer to have my plugins as start, is there a way to make this the global default (and keep on-demand plugins as opt=true)?

[lewis6991]

Now by default all plugins are installed into $XDG_DATA_HOME/nvim/site/pack/packer/opt, however every package in this directory will be loaded during startup unless it has a lazy loading key, or lazy = true. For the most part every package should still behave like a start plugin and this is just an implementation detail that simplifies FS management of plugins.

The differences:

• Plugins are not available in init.lua until packer runs (and loads the packages).
  • For packages like impatient you have need to specify start = true or manually run :packadd impatient.nvim.
• Only the plugins specified via packer.add will be loaded. This makes it so the plugins which get loaded are now a function of what the user provides to packer, as opposed to that plus the filesystem state. This makes packer behave much more similarly to vim-plug, and also makes reducing issues much easier since you don't need to uninstall packages.
• It inflates runtimepath and makes require and autoload a little slower, though I plan to optimize this (within packer or core), and impatient already resolves any slowdown from this anyway.

If I prefer to have my plugins as start,

Is there a specific reason for this?

is there a way to make this the global default (and keep on-demand plugins as opt=true)?

We can provide such an option, but for now I'd like everyone to go with the opt-default method to see how that fairs.

Further, I am also considering dropping use of bram-packages all together, for the main reason that It locks down the implementation and prevents any optimization. Packer has a lot of information about the packages and that information can be weaved into the low level loading operations to speed things up.

[clason]

Plugins are not available in init.lua until packer runs

Nor in plugin/ files, it seems

Is there a specific reason for this?

Yes, pretty much the point above: They just work™️ from plugin/ files, without overhead. With very few exceptions, I don't want on-demand loading, so this simplifies (and modularizes) my config a lot.

Also, a lot of work went in to make bfredlpackages work (specifically for packer!), so it feels churlish not to make use of it.

We can provide such an option, but for now I'd like everyone to go with the opt-default method to see how that fairs.

Well, it broke my config :) So it doesn't fare well for me...

Further, I am also considering dropping use of bram-packages all together, for the main reason that It locks down the implementation and prevents any optimization. Packer has a lot of information about the packages and that information can be weaved into the low level loading operations to speed things up.

I think that would be the logical move here; the current opt-package dance doesn't make much sense if you want to have full control.

(Not sure if that makes Packer v2 the manager for me then, but that need not concern you.)

[lewis6991]

Nor in plugin/ files, it seems

Sorry, I hadn't yet considered this use case. I thought about it a bit, and it should be ok to do this in theory, however it's much more sound to call add() during init.lua since this is when rtp is modified and thus will allow require to work in all plugin/ files. However since packers plugin/ is likely to run first, we might get away with it. I just need a way of detecting when we're in the "source plugin/" section of initialization.

I'll look into it.

Yes, pretty much the point above: They just work™️ from plugin/ files, without overhead. With very few exceptions, I don't want on-demand loading, so this simplifies (and modularizes) my config a lot.

This is probably enough to justify adding a start default option. In my config, instead I just call require'lewis6991.plugins in my init.lua. Same result, just executed before anything in plugin/ which is when you want a package manager to setup.

Also, a lot of work went in to make bfredlpackages work (specifically for packer!), so it feels churlish not to make use of it.

Half the work that went in was figuring out how things should work in the nvim initialization sequence which I'm trying to follow.

The other half was compressing rtp and optimizing globs. I want to take that further and remove the need for globs entirely.

Well, it broke my config :) So it doesn't fare well for me...

It would be useful to know specifically what didn't work.

[wbthomason]

Plugins are not available in init.lua until packer runs (and loads the packages).

How does this interact with impatient caching packer, if impatient can't load until after the first packer require?

Also re: using brampackages/bfredlpackages/"classic" RTP-munging: maybe the best of both worlds would be exposing a finer-grained API for package use in core? That way bfredlpackages still remain and work for cases where they're the right choice (e.g. @clason's basic "just load the damn things" case), but the possibility to do more tricks using information about plugins arises?

This would, of course, require more direct cooperation with core and effectively break v2 for anyone not on latest Neovim, so there are downsides. I do think having this kind of API would be worth it, though - it could also help speed up all the package-based plugin managers by reducing the amount of wheel reinvention that everyone has to do.

[lewis6991]

How does this interact with impatient caching packer, if impatient can't load until after the first packer require?

Impatient will still work since it doesn't need a plugin to be loaded to use its cache. This is arguably a bug, but a bug without negative consequences...unless you're developing a package manager.

For now I've disabled the modpath-cache so I can catch any issues impatient might be masking:

_G.__luacache_config = {
  -- This obscures bugs in packer.nvim and plugin loading order
  modpaths = {
    enable = false,
  }
}

The chunk cache will work fine which provides most of the speed up anyway.

Also re: using brampackages/bfredlpackages/"classic" RTP-munging: maybe the best of both worlds would be exposing a finer-grained API for package use in core?

I don't think we really need an API, don't we already have everything we need? Native packages have a pretty simple implementation, I'm not sure what could/needs exposing. The most we might need is just ways of detecting when we are at certain points in the nvim init sequence.

[wbthomason]

re: API: I was mostly thinking of this:

Half the work that went in was figuring out how things should work in the nvim initialization sequence which I'm trying to follow.

If we had, for example, hooks for the various stages, or an API that could register loads/sourcing for the right stage, etc., this would be a lot simpler. Similarly, something like RTP compression could be built-in to core, and just be a single API call. Or even just batched sourcing/at least an API call for source rather than going through the cmd interface. Nothing too intricate, but exposing some commonly reimplemented functionality for getting plugins loaded.

[lewis6991]

If we had, for example, hooks for the various stages, or an API that could register loads/sourcing for the right stage, etc.,

From my understanding it is just:

1. adjust rtp with all packages
2. source plugin/* for all packages
3. source ftdetect/*
4. source after/plugin/* for all packages

Similarly, something like RTP compression could be built-in to core, and just be a single API call.

RTP compression is just an optimization for start packages. It just allows a directory of packages to take up just one path in rtp, e.g. ~/.data/nvim/site/pack/*/start/*. I'm not sure how we could leverage this further.

Or even just batched sourcing/at least an API call for source rather than going through the cmd interface. Nothing too intricate, but exposing some commonly reimplemented functionality for getting plugins loaded.

nvim__get_runtime() can already does this, it has an undocumented do_source option.

I don't think there's anything particularly slow about vim.cmd.source, and If there is we can probably optimize it (see https://github.com/neovim/neovim/pull/20906), but the gains will be absolutely insignificant compared to the actual sourcing of the file.

[clason]

It would be useful to know specifically what didn't work.

packages were not available in /plugin scripts, so lots of errors of not finding the setup calls.

However since packers plugin/ is likely to run first, we might get away with it.

Is it? I thought plugin directories were traversed alphabetically.

In my config, instead I just call require'lewis6991.plugins in my init.lua.

Sure. But not having to use require for simple lua scripts (and thus avoiding the small but measurable cost) was the whole motivation for my setup.

nvim__get_runtime() can already does this, it has an undocumented do_source option.

Doesn't support globs, unfortunately: https://github.com/neovim/neovim/pull/18426#issuecomment-1119395440 (free to a good home).

[lewis6991]

packages were not available in /plugin scripts, so lots of errors of not finding the setup calls.

Sounds like you have a bunch of other plugin/ files which are running before packer. If that is the problem, then I think the only solution would be to move the call to packer earlier (in init.lua), or we add a start default option.

Is it? I thought plugin directories were traversed alphabetically.

I actually have no idea. I'm just assuming it would just traverse through rtp left-to-right. Need to check.

Sure. But not having to use require for simple lua scripts (and thus avoiding the small but measurable cost) was the whole motivation for my setup.

Can you not make an exception here? If you're running impatient, the cost will be basically nothing, and still very-very small without. On my machine, without impatient a require costs about 0.2ms (1/5000th of a second), with impatient it is 0.025ms (1/40000th of a second).

[clason]

Can you not make an exception here?

I don't know what I will do if/when I have to change things. You asked about why I want a start option and for an explanation of my current setup, and I complied. What you do with this information is up to you.

(On my system every require call is 2ms pure overhead, independently of caching.)

[lewis6991]

Does that 2ms just account for the module lookup or does it also include the sourcing of the file? Sourcing the file should be much more expensive than the lookup.

[lewis6991]

I don't know what I will do if/when I have to change things. You asked about why I want a start option and for an explanation of my current setup, and I complied. What you do with this information is up to you.

☹️

[max397574]

from :h initialization

10. Load the plugin scripts.                    *load-plugins*
    This does the same as the command:  
        :runtime! plugin/**/*.vim
        :runtime! plugin/**/*.lua
    The result is that all directories in 'runtimepath' will be searched
    for the "plugin" sub-directory and all files ending in ".vim" or
    ".lua" will be sourced (in alphabetical order per directory),
    also in subdirectories. First "*.vim" are sourced, then "*.lua" files.

so they are sourced in alphabetical order

[lewis6991]

in alphabetical order per directory

This suggests to me that each file in plugin/* will be sourced alphabetically, but not when traversing rtp.

[wbthomason]

If we had, for example, hooks for the various stages, or an API that could register loads/sourcing for the right stage, etc.,

From my understanding it is just:

1. adjust rtp with all packages

2. source `plugin/*` for all packages

3. source `ftdetect/*`

4. source `after/plugin/*` for all packages

Similarly, something like RTP compression could be built-in to core, and just be a single API call.

RTP compression is just an optimization for start packages. It just allows a directory of packages to take up just one path in rtp, e.g. ~/.data/nvim/site/pack/*/start/*. I'm not sure how we could leverage this further.

Or even just batched sourcing/at least an API call for source rather than going through the cmd interface. Nothing too intricate, but exposing some commonly reimplemented functionality for getting plugins loaded.

nvim__get_runtime() can already does this, it has an undocumented do_source option.

I don't think there's anything particularly slow about vim.cmd.source, and If there is we can probably optimize it (see neovim/neovim#20906), but the gains will be absolutely insignificant compared to the actual sourcing of the file.

Yes - I'm not really making a "oh, C functions will be faster" argument here, but a "it would be better for the ecosystem of Neovim plugin managers if core exposed a more granular set of tools for controlling the stages of plugin loading/performing commonly desired operations (like RTP compression)" argument. In other words, default native packages could become just calls to these API functions in sequence, and if a plugin manager author wants to improve upon that process, they don't need to reinvent all the stages and logic - they can simply call the API around their changes to the process. I think this point is getting too far from packer specifically, though, so I'll leave it for now.