Plan for Colortemplate v3

[lifepillar]

I'd like to share my current vision for medium/long term development of this project, as a note for myself and also to receive feedback. Nothing below is cast in stone.

This plugin has reached some maturity and, given the ratio of open issues vs stars, I'd say that it's quite stable, although the code is a bit of a mess. It has also become somewhat complex. Besides, last weekend I have started to play with Vim9 script and I was pleasantly surprised how far it's got already. Its performance is also striking: I have measured an order of magnitude improvement in the computation of ΔE (one of the mathematical functions heavily used by Colortemplate). So, what I plan to do is:

• Freeze v2.1 feature-wise.
• Modularize the plugin. Turning it into a package might be appropriate: for instance, the style picker could become a plugin by itself (btw, are there plugin managers capable of dealing with packages?)
• Rewrite Colortemplate in Vim9 script. I have already done that (privately) for the test mini-framework, and I have found it pretty easy to do. This would massively help achieving modularization, given the new “import/export” features of Vim9.
• In the process, update the data structure used to represent a color scheme internally. The current structure is the result of a series of very ad hoc patches added over time. To make the plugin easily extendible, I plan to use a purely relational (as in: the Relational Model) schema. I was even thinking to use SQLite for that, but implementing some relational algebra operators is not difficult, so I plan to remain 100% pure Vim script. Why this approach? you may ask. Consider this example: at a certain point during development, I wanted to keep track of “critical” pairs of colors (those that are currently shown in a template's stats). Since I had not foreseen that requirement, I had to add a data structure and modify various parts of the code to maintain such information. In the new purely relational approach I envisage, that would have required only writing a suitable query (given a good data model, of course, but I have that). Another advantage would be that the performance (I estimate) would mostly depend on the efficiency of such relational operators. The first implementation might be trivial and rather inefficient, but it could be improved over time without changing anything else in the code.
• (This might be especially of interest to @romainl and @habamax) I plan to give precedence to a purely declarative (“strict”) subset of the Colortemplate syntax, suitable for color schemes to be included in Vim: no verbatim blocks, no help/auxfiles, and some other restrictions. This would provide more flexibility for generating the output because highlight group definitions in the template could be freely reordered (currently, the order of statements in a template must be kept in the output because verbatim blocks may be interspersed anywhere).
• Over time, I plan to reimplement the whole (“extended”) syntax, for people wanting the full expressive power of Colortemplate.

I think that's all for now. I'll update this thread if something else comes off my mind. Ah, no ETA for now, but don't expect this to be ready tomorrow… or next month 😉

[romainl]

• Freeze: good idea.
• Package: I think the few new plugin managers built on top of the package feature like minpac (?) do support packages but they are not the most prominent. If vim-plug doesn't support it, then that (good) idea is already dead.
• Data structure: please, keep it 100% pure vimscript. SQLite is a fine piece of software but it adds an external dependency, which elevates a little the barrier of entry.
• Strict: +100.

[lifepillar]

SQLite was only an idea that traversed my mind once. I am against introducing overkill dependencies, too.

A point that I have forgot to mention is Neovim. It is never mentioned in the Readme or in the documentation that Colortemplate supports Neovim (except that it may generate color schemes with some additional code for Neovim); however, people seems to assume that, since it works in Vim, it should also work in Neovim. In fact, the current master works in Neovim just fine (barred bugs), with the exception of some unsupported features (the winbar, balloons, the style picker), which are disabled in Neovim.

Last time I asked (towards the dawn of the Neovim project), I was told that Neovim was not a drop-in replacement for Vim, and maintaining compatibility with Vim was not a goal of the project. I think that this still holds true, although I haven't followed Neovim's development closely. What I know is that features have diverged considerably, and it would be too time consuming for me to target both editors, considering that I rarely use Neovim. So, I want to make it clear that:

• I cannot guarantee that Colortemplate v3 will be compatible with Neovim.

[romainl]

So, I want to make it clear that: I cannot guarantee that Colortemplate v3 will be compatible with Neovim.

I see any compatibility between the two ecosystems as accidental. Something that might be useful in some cases but that can't and shouldn't be guaranteed.

[dvogel]

As one of the foremost projects for vim colorschemes, I'd love to get your feedback on a patch I've submitted to vim for making color names easier to deal with. I'm curious whether and how you might see this new v:colornames feature factoring into future development of vim-colortemplate.

[romainl]

@dvogel thank you for mentioning that PR. I made a couple comments over there.

[lifepillar]

@dvogel Thanks for pointing that out. I would certainly take into account anything related to color schemes that finds its way into Vim. I'll follow that PR.

For the record, my vision for v3 has not changed (I did actually make some progress, not published yet) and I am willing to bring the plan forward. I just haven't had much time to dedicate to my pet projects during these months. If my brain doesn't melt in the summer, that might be a good time to speed it up to completion.

[lifepillar]

It's been a long time, but I'd still like to stick to the plan and release this new major version of Colortemplate… eventually. The points I have mentioned above are still valid. I do not plan to keep backward-compability: Colortemplate v2 has proven to be pretty stable, and will still be useable for the time being. So, we are free to start from a clean slate.

The most important thing I'd like to get straight is Colortemplate's syntax, which should be as declarative as possible—while remaining expressive enough—in order to have maximum freedom when generating code. Ideally, reordering template statements should not affect the output (that means, for instance, that I plan to drop support for verbatim blocks).

So, my question to anyone who has used Colortemplate: is there anything that you find unsatisfactory in templates? Is there anything you would change in the current syntax?

[romainl]

My 2 cents…

• I don't find the way Variant: blocks interact with each other very intuitive. For example, having several Variant: gui 256 16 interspersed with other variant blocks makes for some awkward reading and, at least in my case, requires keeping a bit too much "state" in my head for my liking. It may help with repetition, yes, but with some cost.
• I don't like the way the last changed line jumps to the top of the window after a build.

As for the template, here is a quick and dirty proposal:

Pmenu
    none    colorfoo    bold
    none    red         none    256/16

Where…

• the name of the highlight group is on its own line, highlighted with itself to provide a preview
• the first tab-indented line defines the attributes for the default group
• subsequent tab-indented lines define attributes for different variants
• variants are thus defined at one level down, at the highlight group level

[lifepillar]

Thanks for the valuable feedback!

Variant… requires keeping a bit too much "state"

Yeah, I didn't want to mention anything explicitly, but I have never been too satisfied with variants either, although they are extremely flexible, and you nailed the reason why: that kind of “state“ should be avoided.

I like your syntax. I am not fond of significant white space, but it's probably something that can be supported without being enforced.

I also want to keep supporting alternative definitions of highlight groups based on arbitrary conditions. Now that looks something like:

       #if get(g:, 'some_flag', 0)
           Foobar white black underline
       #else
           Foobar white gray none
       #endif

That might become:

Foobar
    white    gray    none
    white   black underline   when get(g:, 'some_flag', 0)

where when would be a new keyword. The current syntax supports nested #ifs, which I see difficult to port into this new syntax. I have only used nested #ifs in Solarized 8, though: probably, not a big deal.

I wonder whether it makes sense to support the equivalent of:

#if some_condition
    Foobar white black
#endif

where Foobar does not have a default. AFAICT, the main use case for this has been to be able to define plugin/filetype highlight groups only if the user explicitly wanted them, in order to avoid messing up colors when switching between color schemes. But I think that Vim fixed that. I can't think of another reason why one would want a highlight group defined only in certain circumstances, without a default.

You don't mention Background, but I know that you don't like dark/light color schemes. I am wondering whether to drop support for those and make Background just a metadata keyword to set at the beginning as either dark or light. That would double the number of some color schemes (e.g., Gruvbox 8 is four color dark/light schemes; they would become eight).

I don't like the way the last changed line jumps to the top of the window after a build.

Can you elaborate? I don't see anything moving after building a color scheme.

[romainl]

I like where the templating is going. Let's get more feedback, though, I don't want an idea I put together on the spot to become a thing so easily.

Here is what I mean with the jump:

But it might be caused by something on my end, I was never annoyed enough to investigate further.

My position about the use of background is that of a hardliner. Using set bg=dark to obtain a color scheme with a dark background is definitely a hack, and I happen to not like it and be vocal about it, but does that mean that tools should prevent to do that? I don't know.

I have three issues with background:

• The point of background is originally to tell Vim, in case it didn't notice, that the terminal background is "dark" so that it can use brighter colors, or that the terminal background is "light" so that it can use less bright colors. It works as a hint for Vim to use brighter or darker colors depending on the terminal background.

  Using it to define a background color is, in my opinion, the opposite of what it is meant to be used for and I distinctly remember a time when people used to the hack from popular third-party color schemes couldn't understand why it didn't work the same outside of that context.

• It is one conditional too many and thus more opportunity for breakage.

• Very few color schemes with "dark" and "light" variants actually feel like a unique color scheme with two variants. Gruvbox, Solarized, and many others feel, to me, like two separate color schemes. Solarized light is not a light version of solarized dark and the other way around, which makes the use of the background even worse.

In my opinion, to keep it with Solarized, the light version you get with :set bg=light should be called "solarized-beach" or something and we should do :color solarized-beach while we should do :color solarized-something to get the "dark" one. No background hacks required, just one template and a build step to produce the two files.

[lifepillar]

Here is what I mean with the jump

That's weird: I cannot reproduce your issue, but I notice that after building, the buffer moves up one line.

just one template and a build step to produce the two files

That's an option. Or keep the one-to-one correspondence between templates and color schemes, use Include for the common parts, and BuildAll to generate all the output files.

[lifepillar]

This is a very tentative syntax for a highlight group definition (by examples):

1. A default definition that unconditionally applies to every variant:

Pmenu none    colorfoo    bold

2. A definition that applies unconditionally only to 256- and 16-color environments:

Pmenu /256/16 none    darkred    bold

3. A default definition for when a variable or constant transparent is defined and has value true (in the following, this is called a discriminator):

Pmenu +transparent true white    none    none

4. A definition that applies only to 256-color environment and only when the discriminator transparent is true:

Pmenu /256+transparent true darkred    none    none

These would be prioritized as follows (from highest to lowest priority): 4 > 2 > 3 > 1 (another possible prioritization might be 4 > 3 > 2 > 1).

The name of the highlight group can be omitted, in which case the value is inherited from a previous line. This allows us to write the above as follows:

Pmenu          none     colorfoo   bold
  /256/16      none     darkred    bold
  +transparent
          true white    none       none
  /256+transparent
          true darkred  none       none

White space does not matter nor does the order of the declarations. The above would be exactly equivalent to, e.g.:

Pmenu                      none     colorfoo   bold
     /256+transparent true darkred  none       none
      +transparent  true   white    none       none        /256/16               none     darkred    bold

The four definitions above would generate code along these lines:

if # has millions of colors
  if transparent
    Pmenu guifg=white      guibg=NONE         gui=NONE     # Version 3
  else
    Pmenu guifg=NONE       guibg=colorfoo     gui=bold     # Version 1
  endif
endif

if # has 256 colors
  if transparent
    Pmenu ctermfg=darkred  ctermbg=NONE       cterm=NONE  # Version 4
  else
    Pmenu ctermfg=NONE     ctermbg=darkred    cterm=bold  # Version 2
  endif
endif

if # has 16 colors
  Pmenu ctermfg=NONE       ctermbg=darkred    term=bold   # Version 2
endif

if # has no colors
  if transparent
    Pmenu term=NONE  # Version 3
  else
    Pmenu term=bold  # Version 1

if # has 16 colors
  Pmenu none    darkred    bold

The above is kind of a worst-case scenario: very few definitions would be so involved, but I'm trying to cover the general case.

[lifepillar]

I forgot to mention: at most one discriminator per highlight group would be allowed—the reason being that different versions of the same highlight group must always be mutually exclusive. When a definition depends on several conditions, they may be encoded. For instance, if a highlight group is controlled by two 0/1 variables g:foo_bold and g:foo_italic, this could be supported by the following (again, tentative) syntax:

#const style = get(g:, 'foo_bold', 0) + 2 * get(g:, 'foo_italic, 0)

String orange black none
  +style
       1 orange black bold
       2 orange black italic
       3 orange black bold,italic

which would be a shorthand for:

String orange black none
String+style 1 orange black bold
String+style 2 orange black italic
String+style 3 orange black bold,italic

[lifepillar]

So, Colortemplate v3 is starting to become a real thing (finally!). There is still much work to be done, but I believe that the most important milestone (getting the syntax right) has been reached. I have collected some notes in a wiki page.

The code is not in a state that can be used for production work, but I would appreciate if you could play with it and provide some feedback, especially on these two aspects:

• the new syntax;
• the form of the generated color scheme (there are two generators: the default one, and an experimental one that can be used by changing one import).

Please use the current Vim master for the best results (you may get all kinds of errors and/or crashes otherwise), and expect bugs™️!

[cideM]

I wanted to ask if your stance with regards to Neovim compatibility has changed in the meantime or of it's still the same, as in: compatibility is accidental but not a goal of this project?

Okay I should have read the Wiki page you linked, sorry.

Backward compatibility is not a goal: older Vim and Neovim are not supported. The syntax, too, is not backward compatible: to use Colortemplate v3 you will have to update your templates. Colortemplate v2 is stable, working and still supported: you may still use it if you want!

[lifepillar]

The current master works in Neovim, and it can generate some Neovim-specific code when the Neovim: yes directive exists in the template. The current release is stable and, being pure Vim script, I don't see why it shouldn't keep working for the time being (unless Neovim doesn't drop some feature or rewrites it in an incompatible way).

Colortemplate v3 is being rewritten in Vim 9 script, which, last time I checked, Neovim did not support. If that changes, it might be possible that the plugin will work in Neovim, too.

I am not closing the door to Neovim, but I am not conditioning my design choices on Neovim either (first and foremost, the choice of using Vim 9 script), because I don't use it and therefore I cannot provide adequate support for it.

[lifepillar]

Update: the core of Colortemplate v3 (v3 branch of this repository) is functional and can be used to generate color schemes using Vim 9 script syntax. You are invited to try it out and provide feedback! I suggest to start with the sample templates in the templates folder. Old and new code are temporarily coexisting: the new code is mostly inside autoload/v3.

All toolbar buttons should work again, except for Stats. I am updating the code that generates the statistics for v3 (I have factored out much of the code into a separate library), so if you have any request regarding those, that's the right moment to ask!

What's still missing: support for ~ (automatic inference of xterm approximations), verbatim blocks (including help file), more backends (at least one to generate legacy Vim script color schemes), better error reporting, some other minor details,

Using the latest Vim version is warmly recommended.

[lifepillar]

Closing this, as the planning phase is well over by now. Colortemplate v3 is fully functional, although it should be considered alpha-level code at the moment. What remains to be done before merging into master:

• move the style picker into a separate plugin;
• remove all the cruft, that is, the legacy Vim script code;
• update the documentation;
• fix bugs and add tests.