feat: introduce `GrammarState`

[antfu]

Motivations

Sometimes, we might want to highlight a partial of the code snippet, which might not necessarily be a full language statement.

For example, if we want to highlight a TypeScript type annotation Pick<MyObject, string>[] we might do:

import { codeToHtml } from 'shiki'

const highlighted = await codeToHtml('Pick<MyObject, string>[]', { lang: 'ts', theme: 'github-dark' })

We will get the result like this:

This isn't accurate. This is because from the grammar, Shiki doesn't know it's a type annotation already.

Instead, we could append let a: to our code to hint that:

import { codeToHtml } from 'shiki'

const highlighted = await codeToHtml('let a: Pick<MyObject, string>[]', { lang: 'ts', theme: 'github-dark' })

Which will now render:

While the highlighting is now correct, we have another challenge: removing the "temporary snippet" we just added from the highlighted HTML.

Solutions

This PR introduces a new concept, GrammarState, with a few APIs around it. GrammarState is a special token that holds the grammar context and allows you to highlight from an intermediate grammar state, making it easier to highlight code snippets.

For example, you can get the grammar state with the getLastGrammarState method, and pass into the grammarState option:

import { createHighlighter } from 'shiki'

const shiki = await createHighlighter({ langs: ['ts'], themes: ['github-dark'] })

const stateTypeAnnotation = shiki.getLastGrammarState('let a:', { lang: 'ts', theme: 'github-dark' })

const highlightedType = shiki.codeToHtml(
  'Pick<MyObject, string>[]',
  {
     lang: 'ts', 
     theme: 'github-dark',
     grammarState: stateTypeAnnotation // <--- this
  }
)

Now Shiki would highlight correctly as it knows to start as the type annotation. You can keep that grammar state object for multiple uses as well.

---

For one-off grammar context shifting, we also provide a shorthand by the grammarContextCode option:

const highlightedType = shiki.codeToHtml(
  'Pick<MyObject, string>[]',
  {
     lang: 'ts', 
     theme: 'github-dark',
     grammarContextCode: 'let a:' // same as above, a temporary grammar state is created internally
  }
)

Credits

Thanks to @benjamincanac for the idea and discussions

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 96.30%. Comparing base (603713d) to head (0953f83).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #712 +/- ## ========================================== + Coverage 96.21% 96.30% +0.09% ========================================== Files 71 72 +1 Lines 6072 6226 +154 Branches 805 829 +24 ========================================== + Hits 5842 5996 +154 Misses 225 225 Partials 5 5 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[orta]

Cool idea