Sunsetting of Lightshow on 1 May 2022

[lildude]

It's with a heavy heart that I'm announcing that it has been decided to sunset Lightshow.

Lightshow will be shutdown on 1 May 2022.

Why?

GitHub is reviewing all services for ownership, maintenance, usage, security risks and costs and sunsetting those that are no longer used or where the "cost" of maintaining and securing the service outweighs the usage and usefulness. Sadly, Lightshow falls into this category due to it being quite a niche service which in turn has incredibly low usage—we're seeing fewer than 30 legitimate page views a day with many days (not just weekends) having zero hits.

GitHub is also slowly moving away from the old Prettylights syntax highlighter which uses the TextMate compatible grammars in favour of a new syntax highlighter called Treelights which uses Tree-sitter grammars.

What do I use instead?

Unfortunately, there isn't another service available for grammar validation and demonstration like Lightshow, and due to licensing reasons surrounding the old Prettylights syntax highlighter, we can't open source Lightshow either.

The closest you can get to seeing how your grammar will appear on GitHub is to use the GitHub Atom theme (most likely to match GitHub's current colouring, thanks @Alhadis) or the GitHub VSCode Theme. The built-in VSCode scope inspector can help you debug grammars. These won't report errors in your grammar, nor will they pick up uses of Oniguruma-specific regexes which aren't compatible with PCRE which Prettylights uses, but it will show you how things will be hightlighted.

When a new grammar is added to Linguist the grammar compiler will validate your grammar and report warnings and any failures. This should catch any problems when new grammars are added or old grammars replaced. Please pay attention to the output from script/add-grammar when adding or replacing a grammar.

This validation is also performed every time we update the grammars used by Linguist when we make a new release. Any regressions will be reported to the upstream grammar repo and the grammar will not be updated in Linguist.

Can I run the compiler against my grammar to validate it before inclusion or updating?

The grammar compiler doesn't currently have support for checking individual grammars without adding them to Linguist first (PRs are welcome). The script/add-grammar will initiate a validation when you add or replace a grammar.

In order to test if an updated grammar will have any problems, you will need to ensure you meet the dependency requirements and follow the "Getting Started" steps from CONTRIBUTING.md, then...

• update just the grammar submodule: git submodule update --remote vendor/grammars/<grammar-dirname>
• update the grammars.yml file and check all the grammars script/grammar-compiler update. This will report all errors so you'll need to pay attention to those specific to the grammar you are checking. Add the -v flag for more verbose output which will also report non-fatal errors and warnings.

You can repeat these two steps until you are happy with the results.

There is no need to open a PR for updated grammars. These will be pulled in as part of the next Linguist release.

Can I submit a tree-sitter grammar for use on GitHub?

No, not at the moment. Linguist does not accept tree-sitter grammars and there is no public repository or location for requests for addition, but GitHub is considering how best to open this up to the public.

GitHub is slowly switching to the tree-sitter grammars from places like those owned by the Tree-sitter organisation, as they become feature complete, or from direct submissions from the specific language communities. If you would like your tree-sitter grammar considered, please use the "Contact GitHub" link on any page on GitHub.

The list of grammars at https://github.com/github/linguist/blob/master/vendor/README.md will be updated as each grammar is switched.

Will Lightshow be open sourced?

No. As mentioned above, the licensing for some of the code used by the Prettylights syntax highlighter prevents us from open sourcing Lightshow.

[Alhadis]

I suppose the onus of developing a replacement tool for previewing TextMate grammars will fall upon me, sooner or later…

GitHub is also slowly moving away from the old Prettylights syntax highlighter which uses the TextMate compatible grammars in favour of a new syntax highlighter called Treelights which uses Tree-sitter grammars.

How worried should I be about all the TextMate grammars that I've written over the years? Knowing that a grammar I maintain could be replaced at any time by a user-submitted tree-sitter grammar has left me somewhat… restless.

The closest you can get to seeing how your grammar will appear on GitHub is to use the GitHub VSCode Theme.

😉 Actually, the closest you can get is an Atom theme I published recently that goes to insane lengths to match the exact appearance of highlighted source code on GitHub (the readme discusses this in more detail).

Colours are extracted from GitHub's front-end using a headless browser, so keeping it up-to-date shouldn't be a hassle (I noticed Lightshow's CSS was badly-outdated, hence my impetus for automating updates to the theme's styling).

[lildude]

How worried should I be about all the TextMate grammars that I've written over the years? Knowing that a grammar I maintain could be replaced at any time by a user-submitted tree-sitter grammar has left me somewhat… restless.

I don't think you need to be too concerned right now. As I understand it—I'm not on the team involved, so this is only supposition based on what I've seen so far—the team is likely to concentrate on the most popular languages first. And even then, I think the Tree-sitter org and the language community supplied grammars (like the Elixir tree-sitter grammar) are likely to take priority over an individual's.

😉 Actually, the closest you can get is an Atom theme I published recently that goes to insane lengths to match the exact appearance of highlighted source code on GitHub (the readme discusses this in more detail).

Oooo, I didn't know about this. I've added it to the OP. Thanks for pointing it out.

[Alhadis]

Oooo, I didn't know about this. I've added it to the OP.

I just noticed atom.io isn't displaying the preview image in the readme. Typical... 😞

I've updated the link to point to the GitHub repository, and added a sidebar link to its page at atom.io instead.

[TheWitheredStriker]

I suppose the onus of developing a replacement tool for previewing TextMate grammars will fall upon me, sooner or later…

@Alhadis I would be highly appreciative of such a project. I'm definitely part of a minority, but I use Lightshow very commonly, both to test forked grammars and to preview consequences of using linguist-override in gitattributes (i.e. for languages with no or insufficient syntax highlighting, where I usually override it to the closest comparable language).

In any case, it's sad to see Lightshow go. But I understand why it is no longer maintainable, as well as why it can't be open-sourced.

[lildude]

Unfortunately, we've had to abruptly bring the sunset date forward due to https://github.blog/2022-04-15-security-alert-stolen-oauth-user-tokens/. Lightshow is no more 😢.

Closing.

[wooorm]

Hi! Just wanted to share that I (not affiliated with GH) made an Open Source version of PrettyLights: https://github.com/wooorm/starry-night. It uses MIT licensed VS Code tools, the grammars from linguist here, to generate markup that works with GitHubs CSS. It’s not exactly the same markup, but it’s probably good enough for folks that want to mimic GH. Maybe this is far-fetched, and maybe not useful because it isn’t exact, but perhaps its possible to open Lightshow if rebased on top of this?

[Alhadis]

@lildude I don't know if this is connected, but GitHub has stopped highlighting languages that use TextMate grammars for highlighting. Tree-Sitter isn't affected, nor are files that haven't been recently updated. Compare:

TextMate: weizhenye/language-vtt

WEBVTT Language: en-US

01:20:37.600 --> 01:20:39.764
<v McCluskey>How's the Italian food in this restaurant?

Tree-Sitter:

throw new HissyFit(this.should.not.be("happening"));

I've been AWOL for the past few weeks, so I'm unsure how long this has been an issue for.

[lildude]

This won't be related at to the sunsetting of Lightshow all. From a quick tests of randomly checking samples, this doesn't appear to affect all Textmate grammars... so far webvtt is the only one I've found affected, including the samples in this repo which haven't changed since you added them. I'm not entirely sure why as the grammar compiled without issues and is definitely in the tarball used by the highlighter. 🤔

[lildude]

@Alhadis I've looked into this by running lightshow locally and it looks like a problem with the grammar:

I thought maybe this could be a problem with it being a CSON file, but other CSON grammars work. The problem also exists with the JSON file the compiler uses.

[Alhadis]

@lildude AFAIC, there doesn't appear to be anything wrong with the WebVTT grammar. I also recall that *.vtt files were once properly highlighted on GitHub, which suggests there's been a change to GitHub's implementation of TextMate grammars.

[lildude]

🤔 lemme check with the highlighter peeps

[lildude]

Checked, and there have definitely been no recent changes to treelights or prettylights that can explain this. I checked the last time PrettyLights had a relevant change and it was back in Aug 2019 when we fixed a bug which would cause huge CPU consumption issues due to never-ending recursion where a begin pattern is not followed by an end. This affects (affected?) TextMate too. We changed to completely ignore these sections to avoid hitting this issue.

I've checked the grammar and sure enough, there's no end pattern for the file_body which is what is pulling in all of the other patterns so effectively everything is being ignored. Adding end: '*$' resolves the issue:

[Alhadis]

I've checked the grammar and sure enough, there's no end pattern for the file_body which is what is pulling in all of the other patterns so effectively everything is being ignored.

So it seems there's a discrepancy between GitHub's {Pretty,Tree}Lights library and Atom's implementation of TextMate-compatible grammars. Until tonight, I was of the belief that the same codebase powered the TextMate highlighting of both environments. Guess I was wrong…

Not to worry though, I've nearly finished writing a replacement WebVTT grammar. I'll submit a PR to replace both this and our grammar for SubRip Text (.srt) files.

[Nixinova]

Lightshow is too useful to not have so I have made my own version here: https://novalightshow.netlify.app/ (https://github.com/Nixinova/NovaLightshow)

[Alhadis]

Lightshow is too useful to not have so I have made my own version here: https://novalightshow.netlify.app/

How did you gain access to PrettyLights? (Or TreeLights, or whatever it's called…) AFAIK, both are closed-source.

[wooorm]

See the source code: https://github.com/Nixinova/NovaLightshow/blob/cdd3b265fdf07d9b7bbd3e1fa860703c70e85fff/index.mjs#L1

I commented about this right here https://github.com/github/linguist/issues/5844#issuecomment-1126737761 but i guess it was snowed under

[Alhadis]

Damn. See, I'm interested in PrettyLights/TreeLights strictly because I'd like to research compatibility differences between it and the various other TextMate-based highlighting systems (Atom, Sublime, VSCode, etc). They're consistent for the most part, but over the years, I've noticed subtle discrepancies that are enough to affect grammars that exploit edge-case behaviours (mainly to circumvent the limitations of the TextMate format). In particular, the logic around newline handling can have a big impact on patterns designed to take priority over another.

It drives me mental, but all I can do is prod GitHub with a stick and see what comes out. 🤷

[TheWitheredStriker]

Lightshow is too useful to not have so I have made my own version here: https://novalightshow.netlify.app/

Incredible! I starred your repository. Thanks for creating this!