Port all templates to Colortemplate v3 syntax

[lifepillar]

This PR updates the syntax of all the templates to Colortemplate v3. I did it so you don't have too 😉

I am not asking to merge this at once. As Colortemplate v3 is still in alpha, this can be used to discuss issues and fix them.

Why upgrade? There are a number of reasons. I will mention just three:

• a more declarative syntax. The Variant directive has been dropped: instead, each highlight group definition has a unique default definition (which must appear first), and zero or more overriding definitions (which may appear anywhere in any order after the corresponding default definition).
• Better performance. Just try!
• A Relational engine. This is an implementation detail, but it's crucially important, as it makes it much easier to enforce integrity constraints. In fact, in a few cases during the porting, I have found a few inconsistencies in the templates (conflicting definitions): please search for FIXME tags.

The generated code is different from v2's, but it is subject to change depending on your feedback. Generating Vim 9 script is also possible, but for now I have rebuilt the colorschemes in legacy Vim script.

Important: I did not check whether the rebuilt colorschemes are visually equivalent to the current ones. It is likely that I've made mistakes.

[clason]

Generating Vim 9 script is also possible, but for now I have rebuilt the colorschemes in legacy Vim script.

We (Neovim) appreciate that! We include these colorschemes as part of Vim's runtime files, and being able to add a new colorscheme from your project is always one of the highlights of porting a runtime file update. I'm really impressed by all the care and effort that you put into these schemes, and I've heard nothing but praise for them! So I just wanted to take the opportunity to say "Thank you!"

[romainl]

I've heard nothing but praise for them!

Funny you say that. Beyond a handful of complaints I have heard nothing positive or negative about our efforts.

[clason]

Then I should have commented here much sooner! (I have meant to for a while.) I can assure you that your efforts are seen and appreciated.

(The bundled colorschemes used to be something of a joke and only really useful as a prank on a coworker; now they are actually useable, and it's even a tough choice which of them to actually use. My personal favorites are habamax, retrobox, lunaperche, and quiet.)

[romainl]

Well thank you. The original offering was indeed in a sad state and it wasn't exactly easy to fix the mess. We hope to be able to keep up as long as possible.

There is still work to do, though:

• review/write a proper documentation, both here and in Vim
• squash remaining highlighting-related bugs and oversights in Vim
• take a hard look at default
• etc.

[lifepillar]

We (Neovim) appreciate that!

Thanks! I'll take the opportunity to raise a question to everyone involved in this project and to Neovim developers if anyone is reading, since in Colortemplate v3 I have dropped Neovim compatibility.

To what extent should people working on this repo care about Neovim?

Not supporting Neovim means cleaner colorscheme's code, easier troubleshooting (one runtime vs two), and the possibility to move the scripts to Vim 9 script. On the other hand, such a move is going to… disappoint Neovim users and likely cause reactions such as this.

The way I see it, Neovim and Vim are diverging more and more, so IMO the wisest choice is to have colorschemes and templates tailored to each system—which is the solution I have adopted with Solarized 8 and Gruvbox 8, where Neovim templates are in a separate branch and are compiled with Colortemplate v2 (which will continue to support Neovim).

Just a few highlights:

• Neovim has a new syntax for scoped highlight groups.
• Neovim has deprecated t_xx settings, so a colorscheme shouldn't rely on t_Co to determine the available colors. See this thread.
• With termguicolors on, Neovim uses gui colors and attributes, but Vim keeps using cterm attributes.
• Neovim does not (currently) support ctermul.

[romainl]

To what extent should people working on this repo care about Neovim?

You already know the answer, don't you? No extent whatsoever.

This project is only concerned with Vim. If the stuff we do here works in Neovim out-of-the-box, then it should be thought of as a happy accident, not a design goal. Just like what is going on in the main repo vim/vim.

You certainly can add Neovim support to colortemplate if you feel like it (that's your project after all) but you should probably put it behind a flag or something if combining Vim and Neovim stuff has any chance of falling apart in Vim.

[clason]

Thanks! I'll take the opportunity to raise a question to everyone involved in this project and to Neovim developers if anyone is reading, since in Colortemplate v3 I have dropped Neovim compatibility.

I didn't even know there was Neovim compatibility before 😅 Can you comment on that?

In any case, I'm mostly concerned with being able to use the finished colorschemes (via Vim) in Neovim, not so much in the Colortemplate script itself (although it definitely looks neat).

To what extent should people working on this repo care about Neovim?

Well, that's obviously up to you; as your stated goal is to be a Vim colorscheme incubator, anything above "not at all" is a nice surprise :)

and the possibility to move the scripts to Vim 9 script. On the other hand, such a move is going to… disappoint Neovim users and likely cause reactions https://github.com/vim/vim/issues/10968.

Yes, that would be pretty much the only thing that would cut us off from your efforts (although there's https://github.com/tjdevries/vim9jit, which might be extendable to cover this). So as long as you feel comfortable sticking to "common vimscript" for these, it will be appreciated.

(I should mention that I feel justified in such an ask -- once, and politely -- since it means you'll have a bigger audience, meaning more people coming with useful bug reports and PRs.)

Everything else you've listed is our problem, and -- so far -- fairly easy to handle manually when I port Vim's runtime updates. And obviously you're not going to add Neovim-only highlight groups to your colorschemes (we handle these by having a reasonable default link to one of the standard groups).

[lifepillar]

I didn't even know there was Neovim compatibility before 😅 Can you comment on that?

Answered in Colortemplate's repo.

[brammool]

Thanks! I'll take the opportunity to raise a question to everyone involved in this project and to Neovim developers if anyone is reading, since in Colortemplate v3 I have dropped Neovim compatibility.

To what extent should people working on this repo care about Neovim?

Whatever the answer is, if you advertise that the colorschemes work with Neovim, you will need to test the colorscheme with both Vim and Neovim. As mentioned, Neovim is making changes in various places, you cannot simply assume that when a colorscheme works for Vim it will also work for Neovim. Once you find a difference you can hopefully "hide" it in a template or add some kind of test, but you do need to keep checking.

• Neovim has a new syntax for scoped highlight groups.

I don't know the details, but it looks very much like what every syntax file has as a list of "hi def link" commands. Something that is recognized as a specific item can be given a specific highlight, but most users will just use the default translation to the limited list of highlight groups used for Syntax highlighting.

If (and that is a big "if") treesitter syntax files are going be used by Vim we might try to use the same names. Then colorschemes made for a specific treesitter supported language could then work both in Vim and Neovim. However, several people have voiced their opinion that treesitter is not a very good choice. But there aren't many alternatives. I'll have to dive into that at some point, once I have sufficient time.

• Neovim has deprecated t_xx settings, so a colorscheme shouldn't rely on t_Co to determine the available colors. See this thread.

Dropping t_xx settings means that quite a few legacy Vim scripts stop working. I thought that Neovim was keeping support for legacy Vim script, but changes like this indicate otherwise. Anyway, Neovim users that run into this should bring up the issue with Neovim. If this happens often enough perhaps they will change their mind.

• With termguicolors on, Neovim uses gui colors and attributes, but Vim keeps using cterm attributes.

For Vim this was decided at some point, and it's not going to change (it would break lots of user's configuration). I suppose Neovim made a different choice because they didn't have to worry about backwards compatibility and thought using "gui" attributes is more logical. Which is true, but then you cause colorschemes to be incompatible...

• Neovim does not (currently) support ctermul.

I suppose adding that to the :highlight command would be easy, but the implementation very much depends on the terminal. I can't find a reference to it in the terminfo manpage. It is implemented for xterm (Vim has this builtin). Kitty does something similar, but I don't see this reflected in the terminfo entry. Other terminals simply do not support a different underline color. Depending on the support makes the colorscheme unusable with quite a few terminals.

-- You are not really successful until someone claims he sat beside you in school.

/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net \\ /// \\ \\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lifepillar]

if you advertise that the colorschemes work with Neovim, you will need to test the colorscheme with both Vim and Neovim.

Absolutely. I think it's fair to say that this is not going to happen in this repo, as @romainl has pointed out. This project focuses only on Vim.

Once you find a difference you can hopefully "hide" it in a template or add some kind of test, but you do need to keep checking.

Yes, templating can act as an interface. Neovim developers can take the templates rather than the colorschemes, and use them to generate their own version of the colorschemes. This is already possible with Colortemplate v2. It will be possible with v3 if someone joins the Colortemplate project to take care of Neovim compatibility, as I am not a Neovim user myself.

Dropping t_xx settings means that quite a few legacy Vim scripts stop working.

AFAICS, they have been made read-only and marked as “deprecated”, but they still exist. But that means, for instance, that the user cannot set &t_Co = 16 to force the use of the underlying terminal colors.

the implementation [of ctermul] very much depends on the terminal. […] Depending on the support makes the colorscheme unusable with quite a few terminals.

That's why ctermul attributes are never present in the colorschemes of this repo. If would be nice if Vim could ignore the special color unless it detects that it is supported.

[brammool]

Dropping t_xx settings means that quite a few legacy Vim scripts stop working.

AFAICS, they have been made read-only and marked as “deprecated”, but they still exist. But that means, for instance, that the user cannot set &t_Co = 16 to force the use of the underlying terminal colors.

I did a simple search for this, but could not find information about it, other than what can be found at ":help t_xx" in http://neovim.io/doc/user/vim_diff.html But there it says "Nvim does not have special t_XX options nor keycodes to configure terminal capabilities.". That is not the same as what you mention, where did you find that?

I haven't seen a rationale for leaving out the t_xx options (I haven't tried searching much though).

the implementation [of ctermul] very much depends on the terminal. […] Depending on the support makes the colorscheme unusable with quite a few terminals.

That's why ctermul attributes are never present in the colorschemes of this repo. If would be nice if Vim could ignore the special color unless it detects that it is supported.

Isn't it? The code doesn't do anything if t_AU isn't set.

-- Although the scythe isn't pre-eminent among the weapons of war, anyone who has been on the wrong end of, say, a peasants' revolt will know that in skilled hands it is fearsome. -- (Terry Pratchett, Mort)

/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net \\ /// \\ \\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[brammool]

I've heard nothing but praise for them!

Funny you say that. Beyond a handful of complaints I have heard nothing positive or negative about our efforts.

For most of this kind of work: "No news is good news". Usually people only complain when they don't like something.

When it works fine then you don't hear anything. Or nobody is using it, that's hard to tell.

-- hundred-and-one symptoms of being an internet addict:

151. You find yourself engaged to someone you've never actually met, except through e-mail.

     /// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net \\ /// \\ \\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lifepillar]

"Nvim does not have special t_XX options nor keycodes to configure terminal capabilities.". That is not the same as what you mention, where did you find that?

You are right, I can find mentions of “t_xx have been deprecated” only in some relatively old issue reports (e.g., https://github.com/neovim/neovim/issues/16832 and https://github.com/neovim/neovim/issues/11883). echo &t_xx does not raise error for any xx but afaics it returns nothing except for t_Co where it returns 256. So, yes, they are basically unsupported, not just deprecated.

No idea about the rationale, except that being compatible with less capable terminals does not seem a Neovim requirement, and “Nvim startup is not blocked on waiting for the UI, including the TUI. So depending on a particular value of &t_Co during startup isn't going to work.” (https://github.com/neovim/neovim/issues/5538#issuecomment-464464644).

Isn't it? The code doesn't do anything if t_AU isn't set.

Ah, I see. I have my terminal misconfigured then…

[clason]

No idea about the rationale, except that being compatible with less capable terminals does not seem a Neovim requirement, and “Nvim startup is not blocked on waiting for the UI, including the TUI. So depending on a particular value of &t_Co during startup isn't going to work.”

That is exactly the rationale. We have instead tried to make terminal feature detection as robust as possible.

[brammool]

"Nvim does not have special t_XX options nor keycodes to configure terminal capabilities.". That is not the same as what you mention, where did you find that?

You are right, I can find mentions of “t_xx have been deprecated” only in some relatively old issue reports (e.g., https://github.com/neovim/neovim/issues/16832 and https://github.com/neovim/neovim/issues/11883). echo &t_xx does not raise error for any xx but afaics it returns nothing except for t_Co where it returns 256. So, yes, they are basically unsupported, not just deprecated.

No idea about the rationale, except that being compatible with less capable terminals does not seem a Neovim requirement, and “Nvim startup is not blocked on waiting for the UI, including the TUI. So depending on a particular value of &t_Co during startup isn't going to work.” (https://github.com/neovim/neovim/issues/5538#issuecomment-464464644).

Related to Neovim having split the UI into a separate process? From the user POV it's still one program though.

It is true that t_Co may have an initial value on startup and change once we know more about the terminal. This happens especially for xterm, which reports the number of colors interactively. The t_Co value is updated later. If it changes and a colorscheme was loaded, it will be reloaded.

So there is no "waiting for the UI" in Vim. The redraw triggered by getting another value for t_Co can be annoying though (especially when debugging and for running tests).

-- hundred-and-one symptoms of being an internet addict:

155. You forget to eat because you're too busy surfing the net.

     /// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net \\ /// \\ \\ sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ /// \\ help me help AIDS victims -- http://ICCF-Holland.org ///

[lifepillar]

I'm closing this PR because many changes have happened in this repo since it was opened, and I've not released an official Colortemplate v3 yet (although the v3 branch is fully functional). The current master of Colortemplate (v2) will be supported for the time being, so there is no hurry to switch—except that v3 is so much better and faster that you might wish you could switch right away :-)

I will consider proposing a translation of the templates of this repo when v3 is officially out.