tango highlight theme displaying incorrect colors after (roughly) Pandoc 2.11

[danbraswell]

The distinctive "tango" highlight theme applied to R changed dramatically or broke sometime around Pandoc 2.11. This is all based on me running pandoc both in the latest Rstudio locally and with an Rmarkdown file at Kaggle. This issue shows up in both.

Here is an example of the tango highlighting as it should be from pandoc 2.9: Note the reddish pipe operator. Very nice highlight style.

Now here is a code exerpt using tango from Pandoc 2.14:

This is very different. It seems either the theme file changed or the mapping changed. One thing noticed digging in theme files is that the R pipe operator "%>%" and the plus operator "+" (as well as others) may have been re-mapped from "Operator" to "SpecialChar".

Regardless of the reason for the change, it would be nice to have tango again displaying the original, and intended, colors.

[jgm]

This is how skylighting tokenizes the R snippet:

[ [ ( NormalTok , "p1 " )
  , ( OtherTok , "<-" )
  , ( NormalTok , " air_visits " )
  , ( SpecialCharTok , "%>%" )
  ]
, [ ( NormalTok , "  " )
  , ( FunctionTok , "group_by" )
  , ( NormalTok , "(visit_date) " )
  , ( SpecialCharTok , "%>%" )
  ]
, [ ( NormalTok , "  " )
  , ( FunctionTok , "summarise" )
  , ( NormalTok , "(" )
  , ( AttributeTok , "all_visitors =" )
  , ( NormalTok , " " )
  , ( FunctionTok , "sum" )
  , ( NormalTok , "(visitors)) " )
  , ( SpecialCharTok , "%>%" )
  ]
, [ ( NormalTok , "  " )
  , ( FunctionTok , "ggplot" )
  , ( NormalTok , "(" )
  , ( FunctionTok , "aes" )
  , ( NormalTok , "(visit_date,all_visitors)) " )
  , ( SpecialCharTok , "+" )
  ]
, [ ( NormalTok , "  " )
  , ( FunctionTok , "geom_line" )
  , ( NormalTok , "(" )
  , ( AttributeTok , "col =" )
  , ( NormalTok , " " )
  , ( StringTok , "\"blue\"" )
  , ( NormalTok , ") " )
  , ( SpecialCharTok , "+" )
  ]
, [ ( NormalTok , "  " )
  , ( FunctionTok , "labs" )
  , ( NormalTok , "(" )
  , ( AttributeTok , "x =" )
  , ( NormalTok , " " )
  , ( StringTok , "\"All visitors\"" )
  , ( NormalTok , ", " )
  , ( AttributeTok , "y =" )
  , ( NormalTok , " " )
  , ( StringTok , "\"Date\"" )
  , ( NormalTok , ")" )
  ]

This is based on the r.xml syntax definition from KDE's syntax highlighting library. No doubt there were changes there that have affected what you're seeing. I'm pretty sure there weren't changes to the tango style itself.

Note that SpecialChar was added to the KDE framework around 2015. At that time I needed to add a color for it to the existing tango style, and I just made it black (probably because I had no idea what it should be). I'd be open to changing it.

[jgm]

(Background in case you're missing it: we use the skylighting library for highlighting, and it generates tokenizers based on KDE's syntax highlighting files.)

Note that the tango theme is not one of the supported KDE themes. It goes way back and I don't even remember where I got those colors in the first place.

You can try the officially supported themes here: https://github.com/KDE/syntax-highlighting/tree/master/data/themes Put it in your working directory and use --style=whatever.theme.

[danbraswell]

Seeing the tokens is very helpful; thank you. How do you generate that token list? I want to do so for a much larger snippet of code. With token lists for the current Pandoc and a Pandoc version before the KDE library change, I could suggest theme color changes to make the highlighting closer to the original.

[jgm]

cabal install skylighting -fexecutable
skylighting --syntax=R --format=native yourfile.r

[danbraswell]

This is an updated tango highlight style that brings back the original colors. Hopefully it can be included in a future release. Thanks for the help.

{
    "text-color": null,
    "background-color": "#f8f8f8",
    "line-number-color": "#aaaaaa",
    "line-number-background-color": null,
    "text-styles": {
        "Other": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Attribute": {
            "text-color": "#204a87",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "SpecialString": {
            "text-color": "#4e9a06",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Annotation": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": true,
            "italic": true,
            "underline": false
        },
        "Function": {
            "text-color": "#204a87",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        },
        "String": {
            "text-color": "#4e9a06",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "ControlFlow": {
            "text-color": "#204a87",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        },
        "Operator": {
            "text-color": "#ce5c00",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        },
        "Error": {
            "text-color": "#a40000",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        },
        "BaseN": {
            "text-color": "#0000cf",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Alert": {
            "text-color": "#ef2929",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Variable": {
            "text-color": "#000000",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Extension": {
            "text-color": null,
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Preprocessor": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": false,
            "italic": true,
            "underline": false
        },
        "Information": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": true,
            "italic": true,
            "underline": false
        },
        "VerbatimString": {
            "text-color": "#4e9a06",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Warning": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": true,
            "italic": true,
            "underline": false
        },
        "Documentation": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": true,
            "italic": true,
            "underline": false
        },
        "Import": {
            "text-color": null,
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Char": {
            "text-color": "#4e9a06",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "DataType": {
            "text-color": "#204a87",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Float": {
            "text-color": "#0000cf",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Comment": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": false,
            "italic": true,
            "underline": false
        },
        "CommentVar": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": true,
            "italic": true,
            "underline": false
        },
        "Constant": {
            "text-color": "#8f5902",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "SpecialChar": {
            "text-color": "#ce5c00",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        },
        "DecVal": {
            "text-color": "#0000cf",
            "background-color": null,
            "bold": false,
            "italic": false,
            "underline": false
        },
        "Keyword": {
            "text-color": "#204a87",
            "background-color": null,
            "bold": true,
            "italic": false,
            "underline": false
        }
    }
}

[jgm]

Perfect, thanks!

[jgm]

I'll transfer this to skylighting and plan to update the style there.