eror msg="syntax highlighting failed (this is a bug, please report it)"

[dongshimou]

• Sourcegraph version:
• Platform information:

Steps to reproduce:

1.clone a repo. 2.docker logs

frontend | t=2023-04-19T20:25:17+0000 lvl=eror msg="syntax highlighting failed (this is a bug, please report it)" filepath=src/runTs.js filetype=javascript repo_name=private.xxxx.com/xx/xxx revision=bdb454fd1d085816cd76920f80157825be5f0739 snippet="\"\\\"use strict\\\";\\nObject.defineProperty(exports, \\\"__esModule\\\", { value: true });\\n/**\"…" error="http://127.0.0.1:9238: unknown error=\"javascript is not a valid filetype for treesitter\" code=\"\""

Expected behavior:

Actual behavior:

If you would like immediate help on this, please email support@sourcegraph.com (you can still create the issue, but there are no SLAs on issues like there are for support requests).

[hfern]

I have the same issue with the single container deployment.

1. Start via docker run --publish 7080:7080 --publish 127.0.0.1:3370:3370 --ulimit nofile=10000:10000 --rm --name sourcegraph --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph -d sourcegraph/server:5.0.2
2. Clone django/django (to pick a random Python project)
3. Go to a python file in the sg UI: https://$host/github.com/django/django/-/blob/setup.py
4. Observe the unhighlighted file and "Syntax-highlighting this file took too long. (TRY AGAIN)"
5. Observer docker log: 05:41:11 frontend | t=2023-04-27T05:41:11+0000 lvl=eror msg="syntax highlighting failed (this is a bug, please report it)" filepath=setup.py filetype=python repo_name=github.com/django/django revision=7d0e56620882c207998a41ac07ec5da572045b31 snippet="\"import os\\nimport site\\nimport sys\\nfrom distutils.sysconfig import get_python_lib\\n\"…" error="http://127.0.0.1:9238: unknown error=\"python is not a valid filetype for treesitter\" code=\"\""

I spot-checked a few file types in the UI:

• Affected and no longer highlighting: Python, Javascript, Java,
• Not affected: Bash, Go, XML

This string appears in the following two places:

1. docker-images/syntax-highlighter/crates/sg-syntax/src/lib.rs
2. docker-images/syntax-highlighter/crates/sg-syntax/src/sg_treesitter.rs

It seems to be all the languages mentioned in #47571. @olafurpg or @tjdevries -- are there any tools so we can debug this on our end?

When I hop into the container and run /usr/local/bin/syntax_highlighter --help, it starts a server instead of dumping help BUT it also helpfully dumps out the languages that are loaded:

bash-5.1# /usr/local/bin/syntax_highlighter --help
...

## Supported file extensions:
...
- Python (`py`, `py3`, `pyw`, `pyi`, `pyx`, `pyx.in`, `pxd`, `pxd.in`, `pxi`, `pxi.in`, `rpy`, `cpy`, `SConstruct`, `Sconstruct`, `sconstruct`, `SConscript`, `pyst`, `pyst-include`, `gyp`, `gypi`, `Snakefile`, `vpy`, `wscript`, `bazel`, `bzl`)
...
🛰  Routes:
   >> (syntect) POST / application/json
   >> (lsif) POST /lsif application/json
   >> (scip) POST /scip application/json
...

[olafurpg]

Thank you for reporting! The root issue is that the single container deployment option is using an outdated version of our syntax-highlighter service

https://sourcegraph.com/github.com/sourcegraph/sourcegraph@4cc162d2654238a52ce0c45257028cbc84a8b95d/-/blob/cmd/server/Dockerfile?L76

Notice that the build is from December 2022. The fix is to send a PR similar to this one here https://github.com/sourcegraph/sourcegraph/pull/42486 to bump the version. I've asked internally for help to somehow automate this version upgrade to prevent it from regressing again in the future.

[hfern]

@olafurpg big thank-you for the info! I have submitted a patch to bump it in #51223.

@dongshimou as a workaround, I just patched the container myself:

Docker image that overrides the /usr/local/bin/syntax_highlighter binary with a newer one (Sourcegraph-Syntax-Patch.Dockerfile):

FROM sourcegraph/server:5.0.2

COPY --from=docker.io/sourcegraph/syntax-highlighter:215692_2023-04-27_5.0-fb61a539c3a1 /syntax_highlighter /usr/local/bin/

Build it:

docker build -f Sourcegraph-Syntax-Patch.Dockerfile -t hfern/sourcegraph/server:5.0.2-syntax .

Then use that image in your docker deployment instead of the usual sourcegraph/server one:

docker run --publish 7080:7080 --publish 127.0.0.1:3370:3370 --ulimit nofile=10000:10000 --rm --name sourcegraph --volume ~/.sourcegraph/config:/etc/sourcegraph --volume ~/.sourcegraph/data:/var/opt/sourcegraph -d hfern/sourcegraph/server:5.0.2

This resolved my issue:

So far I have no issues running this.