Return the grammar state after tokenizing

golok727 commented 3 months ago

Clear and concise description of the problem

codeToTokens to return the grammar state after tokenization.

Alternative

No response

Additional context

No response

Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guide.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Contributes

[ ] If this feature request is accepted, I am willing to submit a PR to fix this issue

antfu commented 3 months ago

What would be the API looks like in your mind?

golok727 commented 3 months ago

What would be the API looks like in your mind?

Including the GrammarState in the TokensResult would be ideal.

Sample usage

class EditorTokensProvider {
// ...
    tokenizeLine(line, state?: GrammarState) {
         const {tokens, grammarState} = highligher.codeToTokens(code, {theme, grammarState: state, lang))

         return {tokens: tokens[0], endState: grammarState }
    }
//...
}

Additionally, could you provide a default static method in the GrammarState to get an initial state?

class GrammarState 
{
 static default(lang, theme) 
{
    return new GrammarState(INITIAL, lang, theme)
}
}

Thank you for this awesome project and for your time!

antfu commented 2 months ago

That would be a breaking change to the function unfortunately. We would need to find another non-breaking API for that

shikijs / shiki