search

alecthomas / chroma

A general purpose syntax highlighter in pure Go
MIT License
4.34k stars 397 forks source link

LRU compiled style cache for the HTML formatter appears to break hugo #945

Closed myitcv closed 7 months ago

myitcv commented 7 months ago

Is there an existing issue for this?

Describe the bug

Upgrading Hugo to use the pseudo-version of chroma implied by https://github.com/alecthomas/chroma/commit/6dd9f269efaa029531b198c3c803f07dc7343c77 appears to reveal a bug in chroma:

git init
git remote add origin https://github.com/gohugoio/hugo
git fetch --depth=1 origin 4f92f949eaf8c9827a758b3caadc672e0335480b
git checkout FETCH_HEAD

# Update to commit immediately prior to problem commit
go get github.com/alecthomas/chroma/v2@v2.12.1-0.20240226204559-898d46707990

# Tidy and test
go mod tidy
go test -count=1 -run TestCommands/gen github.com/gohugoio/hugo

# Update to the problem commit
go get github.com/alecthomas/chroma/v2@v2.12.1-0.20240227060508-6dd9f269efaa

# Tidy and test
go mod tidy
go test -count=1 -run TestCommands/gen github.com/gohugoio/hugo

I expected the above to pass without issue but instead saw:

--- FAIL: TestCommands (0.00s)
    --- FAIL: TestCommands/gen (0.12s)
        testscript.go:558: # Test the gen commands.
            # Note that adding new commands will require updating the NUM_COMMANDS value. (0.116s)
            > env NUM_COMMANDS=42
            > hugo gen -h
            [stdout]
            A collection of several useful generators.

            Usage:
              hugo gen [command]

            Available Commands:
              chromastyles Generate CSS stylesheet for the Chroma code highlighter
              doc          Generate Markdown documentation for the Hugo CLI.
              man          Generate man pages for the Hugo CLI

            Flags:
              -h, --help   help for gen

            Global Flags:
                  --clock string               set the clock used by Hugo, e.g. --clock 2021-11-06T22:30:00.00+09:00
                  --config string              config file (default is hugo.yaml|json|toml)
                  --configDir string           config dir (default "config")
                  --debug                      debug output
              -d, --destination string         filesystem path to write files to
              -e, --environment string         build environment
                  --ignoreVendorPaths string   ignores any _vendor for module paths matching the given Glob pattern
                  --logLevel string            log level (debug|info|warn|error)
                  --quiet                      build in quiet mode
                  --renderToMemory             render to memory (mostly useful when running the server)
              -s, --source string              filesystem path to read files relative from
                  --themesDir string           filesystem path to themes directory
              -v, --verbose                    verbose output

            Use "hugo gen [command] --help" for more information about a command.
            > stdout 'A collection of several useful generators\.'
            > hugo gen doc --dir clidocs
            [stdout]
            Directory clidocs/ does not exist, creating...
            Generating Hugo command-line documentation in clidocs/ ...
            Done.
            > checkfilecount $NUM_COMMANDS clidocs
            > hugo gen man -h
            [stdout]
            This command automatically generates up-to-date man pages of Hugo's
                command-line interface.  By default, it creates the man page files
                in the "man" directory under the current directory.

            Usage:
              hugo gen man [flags] [args]

            Flags:
                  --dir string   the directory to write the man pages. (default "man/")
              -h, --help         help for man

            Global Flags:
                  --clock string               set the clock used by Hugo, e.g. --clock 2021-11-06T22:30:00.00+09:00
                  --config string              config file (default is hugo.yaml|json|toml)
                  --configDir string           config dir (default "config")
                  --debug                      debug output
              -d, --destination string         filesystem path to write files to
              -e, --environment string         build environment
                  --ignoreVendorPaths string   ignores any _vendor for module paths matching the given Glob pattern
                  --logLevel string            log level (debug|info|warn|error)
                  --quiet                      build in quiet mode
                  --renderToMemory             render to memory (mostly useful when running the server)
              -s, --source string              filesystem path to read files relative from
                  --themesDir string           filesystem path to themes directory
              -v, --verbose                    verbose output
            > stdout 'up-to-date man pages'
            > hugo gen man --dir manpages
            [stdout]
            Directory manpages/ does not exist, creating...
            Generating Hugo man pages in manpages/ ...
            Done.
            > checkfilecount $NUM_COMMANDS manpages
            > hugo gen chromastyles -h
            [stdout]
            Generate CSS stylesheet for the Chroma code highlighter for a given style. This stylesheet is needed if markup.highlight.noClasses is disabled in config.

            See https://xyproto.github.io/splash/docs/all.html for a preview of the available styles

            Usage:
              hugo gen chromastyles [flags] [args]

            Flags:
              -h, --help                    help for chromastyles
                  --highlightStyle string   style used for highlighting lines (see https://github.com/alecthomas/chroma)
                  --linesStyle string       style used for line numbers (see https://github.com/alecthomas/chroma)
                  --style string            highlighter style (see https://xyproto.github.io/splash/docs/) (default "friendly")

            Global Flags:
                  --clock string               set the clock used by Hugo, e.g. --clock 2021-11-06T22:30:00.00+09:00
                  --config string              config file (default is hugo.yaml|json|toml)
                  --configDir string           config dir (default "config")
                  --debug                      debug output
              -d, --destination string         filesystem path to write files to
              -e, --environment string         build environment
                  --ignoreVendorPaths string   ignores any _vendor for module paths matching the given Glob pattern
                  --logLevel string            log level (debug|info|warn|error)
                  --quiet                      build in quiet mode
                  --renderToMemory             render to memory (mostly useful when running the server)
              -s, --source string              filesystem path to read files relative from
                  --themesDir string           filesystem path to themes directory
              -v, --verbose                    verbose output
            > stdout 'Generate CSS stylesheet for the Chroma code highlighter'
            > hugo gen chromastyles --style monokai
            [stdout]
            /* Background */ .bg { color:#f8f8f2;background-color:#272822; }
            /* PreWrapper */ .chroma { color:#f8f8f2;background-color:#272822; }
            /* Other */ .chroma .x {  }
            /* Error */ .chroma .err { color:#960050;background-color:#1e0010 }
            /* CodeLine */ .chroma .cl {  }
            /* LineLink */ .chroma .lnlinks { outline:none;text-decoration:none;color:inherit }
            /* LineTableTD */ .chroma .lntd { vertical-align:top;padding:0;margin:0;border:0; }
            /* LineTable */ .chroma .lntable { border-spacing:0;padding:0;margin:0;border:0; }
            /* LineHighlight */ .chroma .hl { background-color:#3c3d38 }
            /* LineNumbersTable */ .chroma .lnt { white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f }
            /* LineNumbers */ .chroma .ln { white-space:pre;-webkit-user-select:none;user-select:none;margin-right:0.4em;padding:0 0.4em 0 0.4em;color:#7f7f7f }
            /* Line */ .chroma .line { display:flex; }
            /* Keyword */ .chroma .k { color:#66d9ef }
            /* KeywordConstant */ .chroma .kc { color:#66d9ef }
            /* KeywordDeclaration */ .chroma .kd { color:#66d9ef }
            /* KeywordNamespace */ .chroma .kn { color:#f92672 }
            /* KeywordPseudo */ .chroma .kp { color:#66d9ef }
            /* KeywordReserved */ .chroma .kr { color:#66d9ef }
            /* KeywordType */ .chroma .kt { color:#66d9ef }
            /* Name */ .chroma .n {  }
            /* NameAttribute */ .chroma .na { color:#a6e22e }
            /* NameBuiltin */ .chroma .nb {  }
            /* NameBuiltinPseudo */ .chroma .bp {  }
            /* NameClass */ .chroma .nc { color:#a6e22e }
            /* NameConstant */ .chroma .no { color:#66d9ef }
            /* NameDecorator */ .chroma .nd { color:#a6e22e }
            /* NameEntity */ .chroma .ni {  }
            /* NameException */ .chroma .ne { color:#a6e22e }
            /* NameFunction */ .chroma .nf { color:#a6e22e }
            /* NameFunctionMagic */ .chroma .fm {  }
            /* NameLabel */ .chroma .nl {  }
            /* NameNamespace */ .chroma .nn {  }
            /* NameOther */ .chroma .nx { color:#a6e22e }
            /* NameProperty */ .chroma .py {  }
            /* NameTag */ .chroma .nt { color:#f92672 }
            /* NameVariable */ .chroma .nv {  }
            /* NameVariableClass */ .chroma .vc {  }
            /* NameVariableGlobal */ .chroma .vg {  }
            /* NameVariableInstance */ .chroma .vi {  }
            /* NameVariableMagic */ .chroma .vm {  }
            /* Literal */ .chroma .l { color:#ae81ff }
            /* LiteralDate */ .chroma .ld { color:#e6db74 }
            /* LiteralString */ .chroma .s { color:#e6db74 }
            /* LiteralStringAffix */ .chroma .sa { color:#e6db74 }
            /* LiteralStringBacktick */ .chroma .sb { color:#e6db74 }
            /* LiteralStringChar */ .chroma .sc { color:#e6db74 }
            /* LiteralStringDelimiter */ .chroma .dl { color:#e6db74 }
            /* LiteralStringDoc */ .chroma .sd { color:#e6db74 }
            /* LiteralStringDouble */ .chroma .s2 { color:#e6db74 }
            /* LiteralStringEscape */ .chroma .se { color:#ae81ff }
            /* LiteralStringHeredoc */ .chroma .sh { color:#e6db74 }
            /* LiteralStringInterpol */ .chroma .si { color:#e6db74 }
            /* LiteralStringOther */ .chroma .sx { color:#e6db74 }
            /* LiteralStringRegex */ .chroma .sr { color:#e6db74 }
            /* LiteralStringSingle */ .chroma .s1 { color:#e6db74 }
            /* LiteralStringSymbol */ .chroma .ss { color:#e6db74 }
            /* LiteralNumber */ .chroma .m { color:#ae81ff }
            /* LiteralNumberBin */ .chroma .mb { color:#ae81ff }
            /* LiteralNumberFloat */ .chroma .mf { color:#ae81ff }
            /* LiteralNumberHex */ .chroma .mh { color:#ae81ff }
            /* LiteralNumberInteger */ .chroma .mi { color:#ae81ff }
            /* LiteralNumberIntegerLong */ .chroma .il { color:#ae81ff }
            /* LiteralNumberOct */ .chroma .mo { color:#ae81ff }
            /* Operator */ .chroma .o { color:#f92672 }
            /* OperatorWord */ .chroma .ow { color:#f92672 }
            /* Punctuation */ .chroma .p {  }
            /* Comment */ .chroma .c { color:#75715e }
            /* CommentHashbang */ .chroma .ch { color:#75715e }
            /* CommentMultiline */ .chroma .cm { color:#75715e }
            /* CommentSingle */ .chroma .c1 { color:#75715e }
            /* CommentSpecial */ .chroma .cs { color:#75715e }
            /* CommentPreproc */ .chroma .cp { color:#75715e }
            /* CommentPreprocFile */ .chroma .cpf { color:#75715e }
            /* Generic */ .chroma .g {  }
            /* GenericDeleted */ .chroma .gd { color:#f92672 }
            /* GenericEmph */ .chroma .ge { font-style:italic }
            /* GenericError */ .chroma .gr {  }
            /* GenericHeading */ .chroma .gh {  }
            /* GenericInserted */ .chroma .gi { color:#a6e22e }
            /* GenericOutput */ .chroma .go {  }
            /* GenericPrompt */ .chroma .gp {  }
            /* GenericStrong */ .chroma .gs { font-weight:bold }
            /* GenericSubheading */ .chroma .gu { color:#75715e }
            /* GenericTraceback */ .chroma .gt {  }
            /* GenericUnderline */ .chroma .gl {  }
            /* TextWhitespace */ .chroma .w {  }
            > stdout '/\* LineHighlight \*/ \.chroma \.hl \{ background-color: #3c3d38 \}'
            FAIL: testscripts/commands/gen.txt:19: no match for `/\* LineHighlight \*/ \.chroma \.hl \{ background-color: #3c3d38 \}` found in stdout

FAIL
FAIL    github.com/gohugoio/hugo        0.144s
FAIL

To Reproduce

See the above steps. I would have expected those steps to succeed, because I assume the change to introduce such a cache was meant to be backwards compatible with no changes required? At least I couldn't see any commentary in the commit message that suggested changes required in calling code.

alecthomas commented 7 months ago

What's the failure? I don't see any output

myitcv commented 7 months ago

What's the failure? I don't see any output

Apologies, now fixed up. I also fixed up the sequence of commands to make clear that it is definitely https://github.com/alecthomas/chroma/commit/6dd9f269efaa029531b198c3c803f07dc7343c77 that broke Hugo in this case.

alecthomas commented 7 months ago

The output changed slightly, in that the styles are now compressed. Are you sure this isn't just a whitespace difference?

alecthomas commented 7 months ago

It looks like that's exactly what the issue is.

alecthomas commented 7 months ago

If that's a real concern, we can add the compress state to the cache key.

myitcv commented 7 months ago

Goodness, I don't know what I did here. I totally missed the line:

/* LineHighlight */ .chroma .hl { background-color:#3c3d38 }

in the actual output. I must have grepped entirely the wrong thing.

Apologies for the noise here, and appreciate the quick response.