Add Semantic Tokens support for aws-lsp-partiql server

XuechunHHH commented 3 months ago

Problem

The current aws-lsp-partiql server does not support Semantic Tokens.

Solution

This update introduces Semantic Tokens support for the aws-lsp-partiql server, enabling syntax highlighting for PartiQL files. The implementation leverages a fallible tree-sitter parser developed by the PartiQL Team.

The parser is compiled into a WebAssembly binary, which is then encoded as a base64 string within a TypeScript file for compatibility with the web-tree-sitter package. Additionally, the tree-sitter.wasmfile from the web-tree-sitter package is similarly compiled into a base64 string and included in a TypeScript file. This setup enables differentiated instantiation in browser and Node.js environments, allowing the server to be used more efficiently as a web worker when assets are loaded from a CDN, compared to direct wasm file bundling.

Testing was conducted on the function wrapped in a standalone runtime in VS Code

Todo

Test the server when wrapped in the web worker runtime with a web-based LSP client.
Resolve the issue where the PartiQL Server could not be launched in the standalone runtime in VS Code following PR#328.
Sync this branch with the current main branch of aws/language-servers after resolving the above issue, as this branch is currently several commits behind.

License

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

akoreman commented 2 months ago

I tested the branch in a web-based LSP client and if we make the changes to support the webworker environment everything seems to work as expected 🚀

akoreman commented 2 months ago

We only include the compiled version of the tree sitter rules (tree-sitter-partiql.wasm), can we also include the files required and some steps or a script to update/expand those tree sitter rules when required?

akoreman commented 2 months ago

Could you also add some automated tests for the semantic tokens to language-server.test.ts and language-service.test.ts to verify that strings of source code get tokenized as expected?

XuechunHHH commented 2 months ago

We only include the compiled version of the tree sitter rules (tree-sitter-partiql.wasm), can we also include the files required and some steps or a script to update/expand those tree sitter rules when required?

That's an interesting proposal. We will definitely do this after open source the tree-sitter-partiql parser by adding a script to pull and compile the latest version of the parser everytime pre-compile the server. Fow now, we will just include the compiled version of parser.

XuechunHHH commented 2 months ago

Could you also add some automated tests for the semantic tokens to language-server.test.ts and language-service.test.ts to verify that strings of source code get tokenized as expected?

Thank you for your suggestion. The tokenization behavior has already been tested within the development of tree-sitter-partiql. Added some automated tests for the semantic tokens to language-server.test.ts and language-service.test.ts to verify the functionality of the onSemanticTokens request handler and the accurate identification of SemanticTokenTypes.

aws / language-servers