collection of known syntax highlighting failures

[slimsag]

The goal of this issue is to collect known syntax highlighting failures from logs on sourcegraph.com.

The rate of failures can be seen here: https://sourcegraph.com/-/debug/grafana/d/syntect-server/syntect-server?orgId=1&from=now-7d&to=now

• worker deaths here are indicating that one of the subprocesses that does highlighting completely stalled and was unresponsive so it had to be killed (we run ~4 worker subprocesses in the Docker image), this is indicating that it tried to highlight a specific file and got completely stuck - a bug in the upstream syntect library code, possibly an incompatibility with one of the syntax definitions we pull in and the stack machine syntect implements not being a 1:1 perfect replica of Sublime's
• panics here are indicating that we tried to highlight a file and the server actually panicked / crashed (rather than worker deaths which imply it became completely unresponsive)

The numbers on this dashboard have been pretty stable over the past 30d, so I don't suspect any regression.

The frontend logs any failed highlighting requests it sends to the service with details on what file/etc to repro the issue, most of them will need to be tracked back to the upstream code https://github.com/trishume/syntect and fixed there.

We can find timeouts (maybe issues, maybe not - only way to tell is to try reproing) by searching frontend logs for syntax highlighting took longer:

{"filepath":"services/horizon/internal/test/scenarios/bindata.go","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/stellar/go","revision":"15de5ffd450223e1bc5abf80575d906dc2eabb6e","severity":"WARNING","snippet":"\"// Code generated by go-bindata. DO NOT EDIT.\\n// sources:\\n// account_merge-core.\"…","t":"2020-11-23T15:50:00.350567747Z"}
{"filepath":"EPPlus/Encryption/EncryptionHandler.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/JanKallman/EPPlus","revision":"55c5ba6169eaa6ae7867fefb36a84cc0b28be85a","severity":"WARNING","snippet":"\"\\ufeff/******************************************************************************\"…","t":"2020-11-23T15:56:34.528219645Z"}
{"filepath":"ui/app/ducks/gas/gas.duck.js","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/MetaMask/metamask-extension","revision":"b3fa1e534e9ca2f10043a58794ae8dd39ec72c5f","severity":"WARNING","snippet":"\"import { uniqBy, cloneDeep, flatten } from 'lodash'\\nimport BigNumber from 'bignu\"…","t":"2020-11-23T16:13:33.583727961Z"}
{"filepath":"src/test/resources/accounts.json","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/NLPchina/elasticsearch-sql","revision":"b12508359929b50cb4527b5173b8afa1f8fa53c3","severity":"WARNING","snippet":"\"{\\\"index\\\":{\\\"_type\\\": \\\"account\\\", \\\"_id\\\":\\\"1\\\"}}\\n{\\\"account_number\\\":1,\\\"balance\\\":39225,\\\"f\"…","t":"2020-11-23T16:29:44.596799065Z"}
{"filepath":"src/Security/Authentication/test/CertificateTests.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/dotnet/aspnetcore","revision":"59e4261f41fecd873617a7673dad24da40f30515","severity":"WARNING","snippet":"\"// Copyright (c) Barry Dorrans. All rights reserved.\\n// Licensed under the Apach\"…","t":"2020-11-23T16:35:01.656655286Z"}
{"filepath":"src/platforms/xamarin.ios/System.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/dotnet/standard","revision":"933433e1a24e972688242da1ea17fada04e80e70","severity":"WARNING","snippet":"\"// Licensed to the .NET Foundation under one or more agreements.\\n// The .NET Fou\"…","t":"2020-11-23T16:40:31.406542073Z"}
{"filepath":"src/platforms/monoandroid/System.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/dotnet/standard","revision":"933433e1a24e972688242da1ea17fada04e80e70","severity":"WARNING","snippet":"\"// Licensed to the .NET Foundation under one or more agreements.\\n// The .NET Fou\"…","t":"2020-11-23T16:40:31.795145175Z"}
{"filepath":"src/WebJobs.Script.Grpc/MessageExtensions/GrpcMessageConversionExtensions.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/Azure/azure-functions-host","revision":"8cd9be240a36de50cfef5961533dbafb6786eb2a","severity":"WARNING","snippet":"\"\\ufeff// Copyright (c) .NET Foundation. All rights reserved.\\n// Licensed under the MI\"…","t":"2020-11-23T16:44:43.668546839Z"}
{"filepath":"ScreenToGif/Webcam/DirectX/CaptureWebcam.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/NickeManarin/ScreenToGif","revision":"4e7bd8414f7e79b4e4597c2540f6aa2860173964","severity":"WARNING","snippet":"\"\\ufeff#region License\\n\\n// -----------------------------------------------------------\"…","t":"2020-11-23T17:07:20.369358259Z"}
{"filepath":"src/MassTransit/Configuration/Metadata/TypeMetadataCache.cs","lvl":"warn","msg":"syntax highlighting took longer than 3s, this *could* indicate a bug in Sourcegraph","repo_name":"github.com/MassTransit/MassTransit","revision":"9217b13879cf226902ffd0b76f54e787ff817a03","severity":"WARNING","snippet":"\"\\ufeffnamespace MassTransit.Metadata\\r\\n{\\r\\n    using System;\\r\\n    using System.Collecti\"…","t":"2020-11-23T17:08:37.816233581Z"}

The really critical ones, however, are the worker timeouts - those indicate one of the server subprocesses died completely. We can find those by searching frontend logs for HSS worker timeout which gives:

{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"src/platforms/xamarin.tvos/System.cs","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/dotnet/standard","revision":"933433e1a24e972688242da1ea17fada04e80e70","severity":"ERROR","snippet":"\"// Licensed to the .NET Foundation under one or more agreements.\\n// The .NET Fou\"…","t":"2020-11-23T16:40:31.415628234Z"}
{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"src/platforms/net461/System.cs","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/dotnet/standard","revision":"933433e1a24e972688242da1ea17fada04e80e70","severity":"ERROR","snippet":"\"// Licensed to the .NET Foundation under one or more agreements.\\n// The .NET Fou\"…","t":"2020-11-23T16:40:31.415736184Z"}
{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"admin/static/js/dropzone.min.js","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/baifendian/harpc","revision":"8cc8128a3b42d1bbbc527daa9edf0ec9ab326468","severity":"ERROR","snippet":"\"!function(){function a(b,c,d){var e=a.resolve(b);if(null==e){d=d||b,c=c||\\\"root\\\";\"…","t":"2020-11-23T16:56:50.495281674Z"}
{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"src/Presentation/Nop.Web/wwwroot/lib_npm/fine-uploader/all.fine-uploader/all.fine-uploader.core.min.js","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/nopSolutions/nopCommerce","revision":"8436c8824c09f2fa18ec924650da2011a207c2ed","severity":"ERROR","snippet":"\"// Fine Uploader 5.16.2 - MIT licensed. http://fineuploader.com\\n!function(global\"…","t":"2020-11-23T16:56:51.056259758Z"}
{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"src/Presentation/Nop.Web/wwwroot/lib_npm/fine-uploader/azure.jquery.fine-uploader/azure.jquery.fine-uploader.min.js","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/nopSolutions/nopCommerce","revision":"8436c8824c09f2fa18ec924650da2011a207c2ed","severity":"ERROR","snippet":"\"// Fine Uploader 5.16.2 - MIT licensed. http://fineuploader.com\\n!function(global\"…","t":"2020-11-23T16:56:51.056539104Z"}
{"error":"http://syntect-server:9238: HSS worker timeout while serving request","filepath":"src/Presentation/Nop.Web/wwwroot/lib_npm/fine-uploader/azure.fine-uploader/azure.fine-uploader.min.js","lvl":"eror","msg":"syntax highlighting failed (this is a bug, please report it)","repo_name":"github.com/nopSolutions/nopCommerce","revision":"8436c8824c09f2fa18ec924650da2011a207c2ed","severity":"ERROR","snippet":"\"// Fine Uploader 5.16.2 - MIT licensed. http://fineuploader.com\\n!function(global\"…","t":"2020-11-23T16:56:51.056546875Z"}

We should look at each of these files in e.g. a dev server and see if we can reproduce them there. If we can, they are bugs in the upstream code github.com/trishume/syntect most likely and we should probably try to track them down there

[varungandhi-src]

Some other hangs which I've been able to reproduce with our fork of syntect:

path=sgtest-megarepo-11c726f/chromium/third_party/blink/perf_tests/speedometer/resources/flightjs-example-app/components/bootstrap/js/bootstrap.js line=143
path=sgtest-megarepo-11c726f/chromium/third_party/chaijs/chai.js line=7479
path=sgtest-megarepo-11c726f/mongo/src/third_party/mozjs/extract/js/src/builtin/intl/CommonFunctions.js line=473
path=sgtest-megarepo-11c726f/grafana/public/vendor/bootstrap/bootstrap.js line=229
path=sgtest-megarepo-11c726f/sourcegraph/client/shared/dev/jest-environment.js line=62

Examples taken from https://github.com/sgtest/megarepo/zipball/11c726fd66bb6252cb8e9c0af8933f5ba0fb1e8d (warning: 2 GB zip file).

Most of these seem to be around a /* comment, with the /* having one or more spaces before it.

I've filed an issue with more details: https://github.com/sourcegraph/syntect/issues/1

---

Results for files in original comment:

• ✅ admin/static/js/dropzone.min.js. No issues on Sourcegraph.com (no highlighting though).
• ✅ all.fine-uploader.core.min.js, works fine. No issues on Sourcegraph.com (highlighting only shows for comments).
• ✅ azure.jquery.fine-uploader.min.js works fine. No issues on Sourcegraph.com (highlighting only shows for comments).
• ✅ azure.fine-uploader.min.js works fine. No issues on Sourcegraph.com (highlighting only shows for comments).

🐢 Slow highlighting for C# (onig = oniguruma which is the default regex engine and supposedly faster, fancy-regex = alternative regex engine in pure rust)

File (size)	M1 Max / Onig	GCP / Onig	GCP / fancy-regex
standard/src/platforms/xamarin.tvos/System.cs (636K)	0.58s	6.1s	2.0s
standard/src/platforms/net461/System.cs (796K)	0.66s	4.4s	1.6s

[jerviscui]

I can't browse the .cs file at all and it doesn't work.

1. No have line number,
2. Then click will get a error Could not find line number.