🚀RFC: New color/style options, diff-so-fancy/diff-highlight emulation

dandavison commented 4 years ago

@0xC0FFEE @Amorymeltzer @CSRaghunandan @Kr1ss-XD @bluz71 @boris-petrov @clnoll @da-x @elbaro @gibfahn @gpressutto5 @jc00ke @lightspeedbriefs @mk12 @natsumi @waldyrious @JNRowe

Hi everyone,

TL;DR: If you are able to run the new delta in master by building from source, that would be really useful and much appreciated. Your current config should just work as before without you changing anything: please let me know if it doesn't.

git pull origin master
cargo build --release
./target/release/delta --help

Thanks very much for your help with delta. I'm opening this issue to describe the new options for configuring colors and styles in the unreleased version of delta in master. To keep the discussion in one place, I've tagged people here who have been helping with delta development and are involved in the various issues related to this topic: any feedback/bug reports/complaints etc you're able to give would be much appreciated. I hope this is OK, let me know if not and please use the Unsubscribe button on the right-hand side of the issues UI to stop notifications from this issue. The notes below will become the release notes for the next release.

Although it seems like a lot of changes, no command-line options have been removed: master should be 100% backwards-compatible.

The master branch has added new options for configuring foreground and background colors, and box/line etc decorations. Every stylable element foo now has two options: --foo-style and --foo-decoration-style. Both accept a "style string". An example of a style string is 'ul bold red "#420000"' meaning "bold, red, underlined foreground (text) color with background color #420000".

So it's basically the same style/color mini-language as is used by gitconfig (and thus by diff-highlight & diff-so-fancy). There are some delta-specific extensions: for example, you use the special color name 'syntax' as a foreground color to request syntax highlighting.

This allows lots of things that weren't possible before: e.g. non-syntax highlighting text colors for code, background colors for unchanged code lines, background color for commit/file/hunk-header lines, and the ability to apply any subset of the 8 available style attributes to text: blink, bold, dim, hidden, italic, reverse, strike, ul (or underline).

Here's an example showing how to use a light theme with a dark terminal background (#153). (This is possible because we can now override the terminal's background colors in more places.)

delta --theme=OneHalfLight
      --minus-style 'syntax auto' --minus-emph-style='syntax auto'
      --zero-style 'syntax white'
      --plus-style 'syntax auto' --plus-emph-style 'syntax auto'
      --hunk-header-style='syntax white' --hunk-header-decoration-style='black box'

And here's diff-so-fancy / diff-highlight emulation:¹ (#177, #169, #193):

delta --minus-style 'red bold' --minus-emph-style 'red bold 52'
      --zero-style 'normal'
      --plus-style 'green bold' --plus-emph-style 'green bold 22'
      --hunk-header-style 'bold syntax' --hunk-header-decoration-style 'bold magenta box'
      --file-style 'bold yellow' --file-decoration-style 'bold yellow underoverline'
      --commit-style 'bold yellow'

delta diff-so-fancy

Here's the relevant section of delta --help:

The old options such as `--minus-color` (meaning the background color) are all still available, and everything is intended to be backwards compatible, but `delta --help` describes the old options as deprecated. Here's a summary of what's changed:		Previously
`--minus-color`, etc	background color	deprecated: use `--foo-style` instead
`--zero-style`	-	New option: "zero" means unchanged lines
`--file-color`, `--commit-color`	foreground color	deprecated: use `--foo-style` instead
`--hunk-color`	decoration foreground color	deprecated: use `--hunk-header-style` instead
`--hunk-style`	{box,underline} decoration	deprecated: synonym of `--hunk-header-decoration-style`
`--file-style`, `--commit-style`	{box,underline} decoration	These now apply to the text. Use `--foo-decoration-style` for the decoration.

_{There's one slightly subtle technicality: --file-style, and --commit-style now configure the text, whereas previously they configured the decoration surrounding the text. But, users' current configs will be using "box" and "underline" in this context to configure the decoration. So, for backwards compatibility, "box" and "underline" (or "ul") are allowed to occur here and are applied to the decoration if they are encountered.}

The changes described here close the following issues:

177 (request for foreground colors enabling behavior like diff-so-fancy / diff-highlight)
169 (request for different styling of lines with and without emph, like diff-highlight/diff-so-fancy)
193 (request for reverse/negative highlighting like diff-highlight)
87, #153 (requests for ability to override terminal's background color)

Related to but does not close yet:

97 (request to retain traditional hunk header within box)
168 (work by @da-x on additional stylable elements; when to allow raw git input with colors through unchanged)

Remaining questions / work needed:

Should future delta offer custom format strings for styling different elements?
When does delta pass on raw colors in the input? (#168)
Should delta take configuration from gitconfig?
Better defaults for delta's within-line highlights (#140)
Need to right-fill with background colors in more places?

To build delta from source:

You'll need to install the rust tools.

git pull origin master
cargo build --release
./target/release/delta --help

¹ The styles are from the example in the diff-so-fancy README.

gibfahn commented 4 years ago

Wow, this is really cool!

I have 9 places in my dotfiles that call delta, so being able to specify this somewhere would be nice.

But also tbh since you fixed all the other issues, the defaults work pretty well for me (except for the theme, which I set with export BAT_THEME="TwoDark").

waldyrious commented 4 years ago

Awesome! I think it would make sense to encode at least diff-highlight (and maybe diff-so-fancy as well, to help with adoption?) as themes.

waldyrious commented 4 years ago

I found what seems to be a bug: specifying "bold" in the box and under/overline decoration style only seems to be affecting the thickness of the lines, but not the color.

boris-petrov commented 4 years ago

@dandavison - that's some amazing work! I tried it and it works like a charm. I'll be keeping an eye for bugs but for now it seems great. Only thing is I had to change the default max-line-distance to 0.8 (not that I understand what this is, just the default didn't catch some changes). Perhaps tweak it a bit for the next release?

Otherwise brilliant work. Thank you!

P.S. Another thing I noticed - using the config for diff-so-fancy you mentioned in:

[interactive]
    diffFilter = ...

And doing a git add -p leads to:

fatal: mismatched output from interactive.diffFilter
hint: Your filter must maintain a one-to-one correspondence
hint: between its input and output lines.

Same as with diff-so-fancy. Using just delta --color-only works. Any ideas why?

waldyrious commented 4 years ago

Shouldn't the "omit" style for decorations omit only the decorations themselves, but keep the content? Or is that the actual intent? (In which case, my question is: how do I deactivate decorations alone but keep the text?)

dandavison commented 4 years ago

I found what seems to be a bug: specifying "bold" in the box and under/overline decoration style only seems to be affecting the thickness of the lines, but not the color.

Thanks @waldyrious. That should be fixed now (ed7c7b8ae36e100dae269aa04d4c1a06f511db3d).

dandavison commented 4 years ago

@boris-petrov, thanks :)

using the config for diff-so-fancy ... and doing a git add -p ...

Git's complaint about "filter must maintain a one-to-one correspondence between its input and output lines." is caused by delta adding file/hunk decorations. We can make git add -p work in diff-so-fancy emulation mode by modifying the command as follows:

--file-decoration-style '' --hunk-header-decoration-style ''

(I'll got through this issue later to add things to documentation.)

dandavison commented 4 years ago

Shouldn't the "omit" style for decorations omit only the decorations themselves, but keep the content? Or is that the actual intent?

Good point @waldyrious. This part of the interface definitely needs more work. Here's a proposal for how it could be changed. I'll probably make a start on implementing this:

`--foo-style=omit`	Nuke the element entirely: display neither element nor decoration.
`--foo-decoration-style=omit`	Just nuke the decoration. Render the text subject to --foo-style as usual.
`--foo-style=raw`	Pass git's raw text through unchanged, including ANSI colors configured in gitconfig, git log --pretty/--format

Additional notes:

Delta doesn't just style the elements comming from git. It also changes them, for example stripping the @@ and line numbers from the hunk header.
So --foo-style=raw will mean two things:
- Don't do this transformation
- Don't add any colors and let git's raw ANSI sequences through
The git docs define empty string to mean "no color effect".
So --foo-style='' could mean:
- Do the transformation to the text
- But don't apply any styling
And --foo-decoration-style='' could be a synonym of --foo-decoration-style=omit.
https://github.com/dandavison/delta/issues/97 requests a decorated hunk-header but without performing the transformation cc @luisdavim. So that would be --foo-style=raw --foo-decoration-style=box
There's related discussion in https://github.com/dandavison/delta/issues/168 about styling the commit section and passing raw git colors through
I can see people trying "none" as a value. If we support that then:
- --foo-style=none should be a synonym of --foo-style=''
- --foo-decoration-style=none should be a synonym of --foo-decoration=omit (to have a box, but with no styling, you just write "box")

how do I deactivate decorations alone but keep the text?

You're right, that doesn't really exist currently. In the proposal above that would be --foo-decoration-style=omit.

waldyrious commented 4 years ago

Everything above sounds reasonable to me!

dandavison commented 4 years ago

@gibfahn thanks!

I have 9 places in my dotfiles that call delta, so being able to specify this somewhere would be nice.

Right. Are you thinking of the comment I made above about whether it would make sense for delta to read from .gitconfig in the future? (I think it could; apart from anything else, configuring strings containing spaces is a bit fraught: e.g. if in ~/.gitconfig we accidentally use outer double quotes instead of outer singles, or if we don't put inner double quotes around a color hex code starting with #, we get a rather obscure error message from git.)

For now, does it help to use a wrapper shell script containing the repeated delta options?

For anyone else reading this who's not sure how to do that, what I mean is something like creating a file called my_delta containing

#!/bin/bash
exec delta --my-option-1 value1 --my-option-2 value2 "$@"

Then doing chmod +x my_delta, and then using /PATH/TO/my_delta in gitconfig and any other places that need to refer to delta with those same options. The bash syntax used there allows you to append additional options, like /PATH/TO/my_delta --my-option-3 value3.

dandavison commented 4 years ago

@waldyrious I've implemented everything proposed in the comment above. The comment has the details but e.g. commit/file/hunk-header elements now support

--thing-decoration-style=none to mean "no decoration but keep the element" (omit means the same as none there)
--thing-style=omit to mean "drop the element entirely"
--thing-style=raw meaning "pass the original colors from git through unchanged" (you can still combine it with a decoration)

waldyrious commented 4 years ago

@dandavison Nice, thanks! Just tested it locally, it works well. I seem to have found a bug, though: if I pass the --file-decoration-style 'box' option, what I get appears to be a combination of the box and the ul styles:

$ git diff | delta-dev --file-decoration-style 'box'

───────────────────────────────────────────┐
renamed: i/diff-test.txt → w/diff-test.txt │
───────────────────────────────────────────┴───────────────────────────────────
⋮

Can you take a look?

(By the way, the detection of a modified file as a rename also seems like a bug)

dandavison commented 4 years ago

Thanks @waldyrious.

By the way, the detection of a modified file as a rename also seems like a bug

Could you post a diff that exhibits incorrect behavior? (We could make a new issue for this; up to you.)

if I pass the --file-decoration-style 'box' option, what I get appears to be a combination of the box and the ul styles

That is actually long-standing delta behavior: for commit and file it draws a horizontal whisker for box. However. Thanks for raising this! I think it's time to make it more consistent and compositional. I'm thinking

--thing-decoration-style='box' => just a box
--thing-decoration-style='box ul' => the box with lower whisker that you saw
--thing-decoration-style='ul ol' => what is currently called "overunderline" (but has never been released)
--thing-decoration-style='box ol' => box with upper whisker (has never existed)
etc

For backwards-compatibility, I'll continue to support box and underline in --thing-style for now, with the original behaviors. But we should all use --thing-decoration-style for them.

dandavison commented 4 years ago

Decoration styles now compose, so we can do things like

--commit-decoration-style='ul ol' => under-and-over-line decoration
--commit-decoration-style='box ul' => box-with-lower-whisker decoration

--commit-decoration-style=box now means just a box.

'box ol' (and 'box ul ol') make sense but are not implemented yet (currently just result in a box): https://github.com/dandavison/delta/issues/214

waldyrious commented 4 years ago

Awesome, this is getting really good! For context, the reason I wanted to remove the ul was because it doesn't play well with resizing the shell (the line wraps). The box, on the other hand, provides a separation without suffering from that issue.

I can now emulate diff-highlight and enhance it:

diff-highlight	delta

I wish there was a way to tell delta to use git's default colors, so that I would only need to pass delta-specific options. For example, --minus-style 'red', --plus-style 'green' and --file-style 'bold yellow' in the example above could be omitted, and maybe even --file-decoration-style 'bold yellow box' could reuse the color of the implicit --file-style, so I'd only need to specify it as --file-decoration-style 'box'. Am I asking for too much? 😅

bluz71 commented 4 years ago

Remaining questions / work needed:

Should delta take configuration from gitconfig?

As a diff-so-fancy user, and soon to be exclusive delta user, ~/.gitconfig makes a lot of sense to store the myriad of styling options. diff-so-fancy already does, and that is easy to remember if a tweak needs to happen 6 months or so later.

Looking forward to the upcoming release incorporating all of this styling work. In my case, I am looking to primarily emulate the diff-so-fancy look augmented by a few extra niceties that delta provides.

Great work @dandavison

mk12 commented 4 years ago

I noticed an issue with emulating diff-so-fancy mode. If the diff simply removes a blank line, you can't tell because delta removes the - sign. I think this is generally a problem for anyone not using a background color for --minus-style.

waldyrious commented 4 years ago

Related to my previous comment

the reason I wanted to remove the ul was because it doesn't play well with resizing the shell (the line wraps)

...I would perhaps like to try adding a background to the file lines, since this doesn't seem to affect line wrapping, and would still provide a full-width visual separator (while also saving two lines per file header!). Is this the sort of thing you were talking about above, @dandavison, when you wrote the following?

Need to right-fill with background colors in more places?

dandavison commented 4 years ago

I would perhaps like to try adding a background to the file lines, since this doesn't seem to affect line wrapping, and would still provide a full-width visual separator. Is this the sort of thing you were talking about above?

Yes, exactly. There's a prototype of that in @da-x's branch linked in https://github.com/dandavison/delta/issues/168. So one possibility is we make background colors in --file-style and --commit-style extend to the full terminal width by default (they will respond dynamically to terminal resizing without wrapping), and then we can use the slightly obscure option --width variable to give the background-color-extends-only-as-far-as-the-text behavior. (That's what --width variable does for other things currently.)

EDIT: I think we would not want background colors to extend to full terminal width when a box decoration is in effect.

dandavison commented 4 years ago

I noticed an issue with emulating diff-so-fancy mode. If the diff simply removes a blank line, you can't tell because delta removes the - sign. I think this is generally a problem for anyone not using a background color for --minus-style.

Thanks @mk12, good point. I think we can say that's a bug: delta should always make changes visible, even if the - sign has been removed. It (There is --keep-plus-minus-markers). Looks like we need to add something like d-s-f's markEmptyLines. Let's discuss this one in #212, which is similar.

dandavison commented 4 years ago

I'm very close, I hope, to releasing the next version of delta (future release won't be like this!). Here are the major changes since the last activity in this issue:

Delta now uses a [delta] section in git config. I'm thinking this will be the primary way of configuring delta from now on. Thanks to everyone who made suggestions about config file solutions (e.g. @gibfahn, @bluz71, @clarfon, @lilyball)
There's a new concept of "features": a feature is a named collection of delta option/value pairs, supplied in git config as e.g. [delta "my-feature"].
--diff-highlight and --diff-so-fancy flags. A diff-so-fancy user coming to delta can get started by using --diff-so-fancy and nothing else.

Here's a quick example of how delta can be configured now. This goes in your usual git config file (delta will look for it in all the same places as git usually does).

[core]
    pager = delta

[interactive]
    diffFilter = delta --color-only

[delta]
    features = diff-so-fancy decorations line-numbers
    syntax-theme = Monokai Extended

[delta "decorations"]
    commit-decoration-style = bold yellow box ul
    file-style = bold 19 ul "#dddddd"
    file-decoration-style = none

[delta "line-numbers"]
    line-numbers-left-format = "{nm:>4}┊"
    line-numbers-right-format = "{np:>4}│ "

You don't have to use features: everything can go under [delta].

The above corresponds to a single command line invocation like this, but we usually won't need to write these out any longer:

delta --features 'diff-so-fancy decorations line-numbers' --syntax-theme 'Monokai Extended' --commit-decoration-style 'bold box ul' ....

If anyone still has energy for helping out, I think the question right now is:

Is there anything about the user interface in master that doesn't seem quite right and that you suspect we're going to end up wanting to change in a backwards incompatible way?

Other changes in master include

--line-numbers gives line numbers
You can use delta file_1 file_2 as a shorthand for diff -u file_1 file_2 | delta
--navigate gives n/N keybindings to jump between files/commits in a large diff or in log -p output.
--word-diff-regexp to change the regex used for tokenization before running the within-line edit inference algorithm
Delta no longer inserts a leading space, so code can be copied directly from the diff. Whitespace errors and blank lines are highlighted with a background color in the first column.

Notes

You can house your delta git config entries in a separate file, using git's standard include mechanisms, e.g.

[include]
    path = ~/.config/delta/delta.conf

flags such as --line-numbers, --diff-highlight, --diff-so-fancy, --navigate etc are shorthands for including the corresponding builtin feature in your features list. In the example above, I could have written
```
[delta]
diff-so-fancy = true
...
```
instead of including diff-so-fancy in the features list. If a user-defined feature has the same name as a built-in feature, it extends the builtin key-value pairs (line-numbers does this above).

A draft of the README for the next version is here.

@waldyrious, your diff-highlight example above can now be achieved like this:

[delta]
    diff-highlight = true
    file-style = bold yellow
    file-decoration-style = 'bold yellow box'

waldyrious commented 4 years ago

Amazing work @dandavison. I love the new configuration modes, and the convenience of the compatibility presets. About the latter, by the way, I'm wondering — is it intentional that e.g. --diff-highlight doesn't exactly replicate all the colors of diff-highlight output? I understand that the UX improvements like removing the +/- signs and simplifying the file headers can be considered strict improvements, but I don't see why the colors would change.

To illustrate what I mean, here's the diff-highlight output:

Here's delta --diff-highlight:

And here's delta --diff-highlight --file-style='bold yellow' --file-decoration-style='bold yellow ul' --hunk-header-style='bold magenta':

I think --diff-highlight should include those colors as well.

(I couldn't get the hunk header to show in bold magenta, btw... what am I doing wrong?)

dandavison commented 4 years ago

Thanks @waldyrious, extremely useful feedback. You're right: diff-highlight was not honoring some keys from color.diff.*. I've fixed that in master. So basically now diff-highlight stays very true to default git output (using raw for commit-style/file-style/hunk-header-style, and thus honoring git config). diff-so-fancy still honors git config, but has its own defaults, and uses delta's hunk header(simplified, syntax highlighted, and in a box).

Default, no color.diff entry in git config:

With color.diff.* git config entries:

[color "diff"]
     meta = bold yellow
     frag = bold magenta

The full details are: there are 3 related built-in features: raw, diff-highlight, diff-so-fancy. raw is extremely similar to default git output and exists mainly for testing. They extend each other as follows (raw is the base, a . means same as column to the left).

	raw	diff-highlight	diff-so-fancy
commit-style	raw	.	bold yellow
commit-decoration-style	none	.	.
file-style	raw	.	color.diff.meta OR 11
file-decoration-style	none	.	bold yellow ul ol
hunk-header-style	raw	.	color.diff.frag OR bold syntax
hunk-header-decoration-style	none	.	magenta box
minus-style	color.diff.old OR red	.	.
plus-style	color.diff.new OR green	.	.
minus-non-emph-style	minus-style	color.diff-highlight.oldNormal OR minus-style	.
minus-emph-style	minus-style	color.diff-highlight.oldHighlight OR reverse minus-style	color.diff-highlight.oldHighlight OR bold red 52
plus-non-emph-style	plus-style	color.diff-highlight.newNormal OR plus-style	.
plus-emph-style	plus-style	color.diff-highlight.newHighlight OR reverse plus-style	color.diff-highlight.newHighlight OR bold green 22
keep-plus-minus-markers	true	false	.

I couldn't get the hunk header to show in bold magenta, btw... what am I doing wrong?

That's weird, I copied your command line and couldn't reproduce it. Can you try again?

waldyrious commented 4 years ago

Interesting. If I pipe the output of git's diff to delta (rather than setting delta as git's pager), the --diff-highlight option doesn't seem to pick up the file and hunk header colors:

(I also tested with git --no-pager diff instead of git -c "core.pager = diff-highlight" diff in the second command, but thee results were the same.)

dandavison commented 4 years ago

If I pipe the output of git's diff to delta (rather than setting delta as git's pager), the --diff-highlight option doesn't seem to pick up the file and hunk header colors:

Ah, right. That occurs because (a) git by default only outputs colors when the output is going to a terminal, and (b) delta --diff-highlight is implemented by using raw as the style string for the commit/file/hunk-header elements. (raw means delta passes on the ANSI color escape sequences unchanged from its input on those lines). An alternative would be for delta to not use raw and re-apply the color by consulting the value in git config. However, currently, that would mean delta would mess with (simplify) the hunk-header, and I was thinking of leaving diff-highlight very bare-bones by default. Anyway...details! We can carry on tweaking these things in the future, as long as we're happy with the outline currently. And hopefully people won't be explicitly piping git to delta much (for diff -u, there's delta file_1 file_2 now)

dandavison commented 4 years ago

OK, thanks very much for all the help everyone. 0.2.0 is released now!

From the release notes (https://github.com/dandavison/delta/releases/tag/0.2.0):

The highlights are:

To configure delta's appearance, you now use "styles" instead of background color names. A style is a string like red bold ul "#ffeeee" that specifies foreground color, background color, and attributes. See https://github.com/dandavison/delta#choosing-colors-styles.
Delta now reads configuration from a [delta] section in git config, and this is the recommended way to configure delta. See https://github.com/dandavison/delta#configuration
Delta now has a line numbers feature, thanks to @clnoll. See https://github.com/dandavison/delta/pull/190, https://github.com/dandavison/delta/pull/219
Delta now has --diff-highlight and --diff-so-fancy emulation modes (thanks @waldyrious for much testing)

bluz71 commented 4 years ago

Outstanding work.

I just fully transferred over to delta from diff-so-fancy. After some configuration hassles I now have a setup that is now superior to what I was running previously. Many thanks.

One minor point, much of the documentation states underoverline as an attribute, however, that no longer works, instead one must define ul ol. I recommend updating the documentation.

Best regards.

dandavison / delta