search

vsce-toolroom / vscode-textmate-languageservice

Language APIs and support features from Textmate tokenization in Visual Studio Code.
Other
15 stars 0 forks source link

readme

vscode-textmate-languageservice

🎉 This package has been adopted by the vsce-toolroom GitHub collective.

This package is in LTS mode & the Textmate technology is superseded by the tree-sitter symbolic-expression parser technology, as used in vscode-anycode.

Language service providers & APIs driven entirely by your Textmate grammar and one configuration file.

To use the API methods and tokenization / outline services, you only need a Textmate grammar. This can be from your extension or one of VS Code's built-in languages.

In order to properly generate full-blown language providers from this module, the Textmate grammar must also include the following features:

Installation

npm install vscode-textmate-languageservice

Browser support:

Advisory:

This package is stable with browser compatibility (1.1.0). But I recommend you watch out for tree-sitter native integration into vscode (issue). Maintainable & with faster retokenization, it is a Holy Grail ...

Whereas this package depends on a well-written Textmate grammar and is a band aid of sorts.

If there is native vscode support for the language, find a Tree-sitter syntax online then suggest it in an Anycode issue. Otherwise, please open an issue on the community-maintained Treesitter syntax highlighter extension and someone might deal with it.

Setup

Example language extension manifest - ./package.json:

{
    "name": "lua",
    "displayName": "Textmate language service for Lua",
    "description": "Lua enhanced support for Visual Studio Code",
    "version": "0.0.1",
    "publisher": "",
    "license": "",
    "engines": {
        "vscode": "^1.51.1"
    },
    "categories": [
        "Programming Languages"
    ],
    "contributes": {
        "languages": [{
            "id": "lua",
            "aliases": ["Lua"],
            "extensions": [".lua", ".moon", ".luau"],
            "configuration": "./language-configuration.json"
        }],
        "grammars": [{
            "language": "lua",
            "scopeName": "source.lua",
            "path": "./syntaxes/Lua.tmLanguage.json"
        }]
    }
}

Configuration

Create a JSON file named textmate-configuration.json in the extension directory. The file accepts comments and trailing commas.

If you only want to use the document and/or tokenization services, you can skip creating the file!

Textmate configuration fields:

Configuration examples

Template for textmate-configuration.json file:

{
  "assignment": {
    "single": "",
    "multiple": "",
    "separator": ""
  },
  "declarations": [],
  "dedentation": [
    "keyword.control.elseif.custom",
    "keyword.control.else.custom"
  ],
  "exclude": "{.modules,.includes}/**",
  "indentation": {
    "punctuation.definition.comment.begin.custom": 1,
    "punctuation.definition.comment.end.custom": -1,
    "keyword.control.begin.custom": 1,
    "keyword.control.end.custom": -1
  },
  "punctuation": {
    "continuation": "punctuation.separator.continuation.line.custom"
  },
  "markers": {
    "start": "^\\s*#?region\\b",
    "end": "^\\s*#?end\\s?region\\b"
  },
  "symbols": {
    "keyword.control.custom": 2,
    "entity.name.function.custom": 11
  }
}

An example configuration file that targets Lua:

{
  "assignment": {
    "single": "meta.assignment.variable.single.lua",
    "multiple": "meta.assignment.variable.group.lua",
    "separator": "punctuation.separator.comma.lua"
  },
  "declarations": [
    "meta.declaration.lua entity.name",
    "meta.assignment.definition.lua entity.name"
  ],
  "dedentation": [
    "keyword.control.elseif.lua",
    "keyword.control.else.lua"
  ],
  "exclude": "{.luarocks,lua_modules}/**",
  "indentation": {
    "punctuation.definition.comment.begin.lua": 1,
    "punctuation.definition.comment.end.lua": -1,
    "keyword.control.begin.lua": 1,
    "keyword.control.end.lua": -1
  },
  "markers": {
    "start": "^\\s*#?region\\b",
    "end": "^\\s*#?end\\s?region\\b"
  },
  "symbols": {
    "keyword.control.lua": 2,
    "entity.name.function.lua": 11
  }
}

Usage

TextmateLanguageService

The package exports a default class named TextmateLanguageService.

The library defaults to core behaviour when figuring out which scope name to use - last matching grammar or language wins. If the context parameter is supplied, the extension will first search contributions from the extension itself.

Language extension

Extension code sample - ./src/extension.ts:

import TextmateLanguageService from 'vscode-textmate-languageservice';

export async function activate(context: vscode.ExtensionContext) {
    const selector: vscode.DocumentSelector = 'lua';
    const textmateService = new TextmateLanguageService(selector, context);

    const foldingRangeProvider = await textmateService.createFoldingRangeProvider();
    const documentSymbolProvider = await textmateService.createDocumentSymbolProvider();
    const workspaceSymbolProvider = await textmateService.createWorkspaceSymbolProvider();
    const definitionProvider = await textmateService.createDefinitionProvider();

    context.subscriptions.push(vscode.languages.registerDocumentSymbolProvider(selector, documentSymbolProvider));
    context.subscriptions.push(vscode.languages.registerFoldingRangeProvider(selector, foldingRangeProvider));
    context.subscriptions.push(vscode.languages.registerWorkspaceSymbolProvider(workspaceSymbolProvider));
    context.subscriptions.push(vscode.languages.registerDefinitionProvider(selector, peekDefinitionProvider));
};

Tokenization

Extension code sample - ./src/extension.ts:

import TextmateLanguageService from 'vscode-textmate-languageservice';

export async function activate(context: vscode.ExtensionContext) {
    const selector: vscode.DocumentSelector = 'custom';
    const textmateService = new TextmateLanguageService('custom', context);
    const textmateTokenService = await textmateService.initTokenService();
    const textDocument = vscode.window.activeTextEditor!.document;
    const tokens = textmateTokenService.fetch(textDocument);
};

NB: If you would like to:

You can use the custom "textmate-languageservice-contributes" property in package.json:

{
    "textmate-languageservice-contributes": {
        "languages": [{
            "id": "typescript",
            "aliases": ["TypeScript"],
            "extensions": [".ts", ".tsx", ".cts", ".mts"]
        }],
        "grammars": [{
            "language": "typescript",
            "scopeName": "source.ts",
            "path": "./syntaxes/TypeScript.tmLanguage.json"
        }]
    }
}

API methods

Usage (example is for getting the token at the current cursor position):

const { getScopeInformationAtPosition } = TextmateLanguageService.api;

const editor = vscode.window.activeTextEditor;
const document = editor.document;
const position = editor.selection.active;

const token = await getScopeInformationAtPosition(document, position);

getScopeInformationAtPosition

getScopeInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise<TextmateToken>

Get token scope information at a specific position (caret line and character number).

getScopeRangeAtPosition

getScopeRangeAtPosition(document: vscode.TextDocument, position: vscode.Position): vscode.Range;

Get matching scope range of the Textmate token intersecting a caret position.

getTokenInformationAtPosition

getTokenInformationAtPosition(document: vscode.TextDocument, position: vscode.Position): Promise<vscode.TokenInformation>;

VS Code compatible performant API for token information at a caret position.

getLanguageConfiguration

getLanguageConfiguration(languageId: string): LanguageDefinition;

Get the language definition point of a language mode identifier.

getGrammarContribution

getGrammarConfiguration(languageId: string): GrammarLanguageDefinition;

Get the grammar definition point of a language mode identifier.

getLanguageContribution

getLanguageConfiguration(languageId: string): LanguageDefinition;

Get the language configuration of a language mode identifier.

getContributorExtension

getContributorExtension(languageId: string): vscode.Extension<unknown> | void;

Get the VS Code Extension API entry of the extension that contributed a language mode identifier.

Use Oniguruma WASM buffer

This is the vscode-oniguruma build of Oniguruma written in C, compiled to WASM format with memory hooks to V8.

This is not streaming 🙁 but vscode libs must bundle WebAssembly deps so as to support web ecosystem.

import TextmateLanguageService from 'vscode-textmate-languageservice';
const onigurumaPromise = TextmateLanguageService.utils.getOniguruma();