Open MaskRay opened 4 years ago
As the field positionEncoding
indicates this will be something that is customizable in LSIF (supporting more encodings than utf-16
only).
Since this has nothing to do with the upcoming 3.16 LSP spec I move it to on deck.
From the lengthy discussions in https://github.com/microsoft/language-server-protocol/issues/376 , a lot of language servers and non-JavaScript based clients have been inconvenienced due to the UTF-16 position counting. In Language Server Index Format, which does not have the backward compatibility burden, we should probably move away from UTF-16 which is considered inferior by a majority of developers. As the recommended default, both UTF-32 code points and UTF-8 bytes can work but it seems that code points are convenient for most implementations.
When opening a non UTF-8 encoded file, an implementation may likely represent it with a sequence of code points. I believe most implementations have convenient APIs to count code points but counting UTF-8 bytes may require more transcoding.
A survey on the Position.character offsets encoding supported by the clients: