Language Server (LSP)

[siefkenj]

pretext-tools should implement Language Server Protocol. Any features implemented in LSP can be used in any editor that speaks LSP (vscode, vim, emacs, etc.).

• A sample implementation is here: https://github.com/microsoft/vscode-extension-samples/tree/main/lsp-sample
• A guide is here: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide

[oscarlevin]

I agree this is an important step to take eventually.

Any thoughts on whether this should be implemented separately from the pretext-cli gui features for vscode? That is, should there be two extensions, one an LSP and the other this current pretext-tools?

[siefkenj]

The answer is yes and no!

The way Microsoft's example works is that the vscode extension runs the lsp in a separate process. With some work, it would be possible to, say, have python also run the lsp independently of vscode (provided node was around). In my branch, I've created a new compile endpoint, and there will be out/extension.js as well as out/lsp-server.js. The extension sets up a server and talks to it as it needs. This can happen on top of other functionality that the extension provides.

Does that answer the question, or did I miss something?

[oscarlevin]

Have you seen this: https://github.com/eclipse/lemminx? Apparently it can be extended to use your own language. I don't understand much of the documentation though.

It might also be worth looking at vscode-xml to see how they implement stuff.

[siefkenj]

I hadn't heard of it, but it is written in Java. I don't think we want another dependency...

[oscarlevin]

Interesting. The vscode-xml plugin no longer requires java. I don't know what that means for "extending" as they discuss. However, I wonder whether it would be better to just use the current vscode-xml to handle the language parts, and set up pretext-tools to specify the right schema for that plugin.

I think this means abandoning the pretext language, and resetting pretext-tools to use xml as the language (with a setting that makes .ptx files register as xml).

[siefkenj]

Interesting that they say it doesn't need Java. I might have to poke around a bit more, because it certainly seems like it's calling out to Java.

I think we should keep ptx separate from general XML. Otherwise I don't think we'll be able to get features like jump to definition for references and such. Also, this way we'll have more control over autocomplete. The down side is it will be more work...

I've been playing around and I have a bunch of stuff working. I'll make a PR soon, but here's a screenshot:

Notice the autocompletion for file names :-). The file names are underlined and are links you can click. Also, in the lower left there is an "outline" of the document that lists all the builds.

I've only worked on making the project.ptx file work nicely. It's mostly an experiment at this point, but it might already be useful to some.

[angelozerr]

Have you seen this: https://github.com/eclipse/lemminx? Apparently it can be extended to use your own language. I don't understand much of the documentation though.

Hi I'm working on LemMinx project and if you have any questions, don't hesitate to ask me. It is a LSP language server for XML written in Java. This language server is used in several LSP client:

• vscode with vscode-xml
• Eclipse IDE in WWD
• sublime with https://github.com/sublimelsp/LSP-lemminx
• emacs https://emacs-lsp.github.io/lsp-mode/page/lsp-xml/
• vim https://github.com/neovim/nvim-lspconfig/blob/master/doc/server_configurations.md#lemminx
• and other like https://github.com/fannheyward/coc-xml

I hadn't heard of it, but it is written in Java. I don't think we want another dependency...

It was true before version of 1.5.0, but today to avoid having the Java Runtime dependency, we provide a binary of lemminx, so it doesn't require install of Java.

Interesting. The vscode-xml plugin no longer requires java. I don't know what that means for "extending" as they discuss.

You can extend it by implementing in Java some completion, codelens, etc participant to manage your custom codelens, participant for your custom language. BUT if you do that, it will require Java install because binary is not extensible. We could discuss about this problem together if you wish (perhaps we could embed PretText particpants in LemMinx?)

However, I wonder whether it would be better to just use the current vscode-xml to handle the language parts, and set up pretext-tools to specify the right schema for that plugin.

It seems that your pretext file don't declare the schema, I suggest that you use file association https://github.com/redhat-developer/vscode-xml/blob/main/docs/Validation.md#XML-file-association-with-xsd

I think this means abandoning the pretext language, and resetting pretext-tools to use xml as the language (with a setting that makes .ptx files register as xml).

Indeed it is the idea, but it is not possible with vscode-xml, but I'm aware to provide this feature.

[angelozerr]

In other words I think you need https://github.com/redhat-developer/vscode-xml/issues/589#issuecomment-908360250 to:

• say that ptx belongs to xmllanguage
• contribute with a file association which maps *.ptx with your schema (toavoid declaring the association in the xml)

[angelozerr]

Please note too that vscode-xml supports

• XML Catalog
• RelaxNG (XML and compact syntax) support

We provide a basic support with XInclude (we will release that this week but you can test it with prerelease https://github.com/redhat-developer/vscode-xml#contributing

and Surround with Tags, Comments, CDATA

[siefkenj]

@angelozerr Thanks for introducing yourself. vscode-xml looks really cool! Do you know if vscode is happy with multiple language servers running at the same time? It might be easiest to have vscode-xml handle the xml-oriented tasks and then have a separate language server for pretext-specific things like goto definition and refactoring, etc.

[angelozerr]

@angelozerr Thanks for introducing yourself.

You are welcome!

vscode-xml looks really cool!

Thanks!

Do you know if vscode is happy with multiple language servers running at the same time? It might be easiest to have vscode-xml handle the xml-oriented tasks and then have a separate language server for pretext-specific things like goto definition and refactoring, etc.

Yes it works perfectly. We did that for managing Qute template https://github.com/redhat-developer/vscode-quarkus/blob/master/docs/qute/TemplateSupport.md which is an HTML file with some custom directive.

We enjoy with the full support of vscode HTML (ex : HTML completion) and our Qute template (ex : Qute template directive), but it will require https://github.com/redhat-developer/vscode-xml/issues/589#issuecomment-908360250 to say that ptx language belongs to xml language.

[oscarlevin]

Nice to connect with you @angelozerr. This sounds really great. I'm especially interested in your comment:

In other words I think you need https://github.com/redhat-developer/vscode-xml/issues/589#issuecomment-908360250 to: say that ptx belongs to xml language contribute with a file association which maps *.ptx with your schema (toavoid declaring the association in the xml)

From what I gather, these are things that cannot be done yet? Of course we can document how someone could set up the file association inside their settings when working on pretext documents (and set this automatically for a Github Codespaces template we are developing). That takes care of the schema. Currently pretext-tools contributes a new language (pretext) triggered on .ptx file extension. So that means that vscode-xml will not recognize these files as xml. Or is there a setting in vscode-xml that allows a user to say what languages it should apply to?

[angelozerr]

In short answer it is not possible without https://github.com/redhat-developer/vscode-xml/issues/589#issuecomment-908360250

The main reason is that you define a new language and vscode doesnt manage the capability to say that your language extends xml language.

When you start a language client you define a documentSelector with a list of languageId like you did iI think with your language server.

We could add pretext in vscode-xml when we create documentSelector to start the language client but we would like to avoid doing that because it is too specific.

The main idea is to say that pretext language must be managed with our vscode xml language client by just declaring that in package.json

By doing that you will benefit with all features from vscode-xml like syntax validation, Quick fix etc in ptx files.

If you are really interested with that please add à comment in the vscode-xml issue.

For the xml completion validation based on schema, you will need to declare file association in the user settings but the clean mean should be to declare this file association in package.json

Where can I find your schema?

[angelozerr]

@siefkenj @oscarlevin I created the PR https://github.com/redhat-developer/vscode-xml/pull/839 to contribut eto vscode-xml with a custom language.

Once the PR will be merged, you will just declare that pretext language should contribute to vscode-xml in package.json https://github.com/redhat-developer/vscode-xml/blob/d6371e3939f5c729b8062ca6d94ab62821dcbdba/docs/Extensions.md#custom-language

It will provide all features like XML syntax validation, autorename, etc

To benefit with RelaxNG validation, an dcompletion you will need to declare in each settings project

"xml.fileAssociations": [
      {
         "pattern": "**/*.ptx",
         "systemId": "path/to/pretext.rng"
      }
   ]

[oscarlevin]

This is fantastic. Can't wait to try it out.

[angelozerr]

I created a PR https://github.com/oscarlevin/pretext-tools/pull/17

[angelozerr]

I add a demo in https://github.com/oscarlevin/pretext-tools/pull/17

[angelozerr]

I think we should keep ptx separate from general XML. Otherwise I don't think we'll be able to get features like jump to definition for references and such.

I have worked about this feature which is requires for TEI and docbook. You can manage those reference easily with settings https://github.com/redhat-developer/vscode-xml/blob/48010206916f2a5ff95f204d6012d8cd7b532d68/docs/Features/XMLReferencesFeatures.md. It is an experimental feature, so please create issues if you need more features.

For the moment references works to reference an attribute (from) to an another attribute (to), but I have the intention to improve it to manage the reference to reference files, dir, etc like you need and you did with your LSP language server.

[angelozerr]

If you install prerelease, you can play with this feature.

Add in your settings.json

"xml.references": [
    {
        "pattern": "**/*.ptx",
        "expressions": [
            {
                "from": "xref/@ref",
                "to": "@xml:id"
            }
        ]
    }
]

and after that you can enjoy with completion, definition, higlight with PreText references:

I have the intention to improve again this reference support to manage:

• find references
• codelens which whill show number of references of a given @xml:id
• rename an @xml:id value to rename all referenced attributes
• manage file, dir for references instead a given attribute (to)
• validation to check if referenced id exists.

With a simple xml.references settings,you will benefit with those all support. For themomemnt, only completion,definition, highlight is supported. Don't hesitate to create issues if you need one of issue that I have in my mind, or if you need another support.

[angelozerr]

@siefkenj I noticed that your LSP server supports formatter. vscode-xml provides an advanced formatter https://github.com/redhat-developer/vscode-xml/blob/main/docs/Formatting.md. I wonder if you think this formatter is enough for you, or if you need another improvement,fixes. Don't hesitate to create any issues to improve it. Thanks!

[angelozerr]

I think we should keep ptx separate from general XML. Otherwise I don't think we'll be able to get features like jump to definition for references and such.

If you speak about your xref element, vscode-xml can support that with settings. See https://github.com/oscarlevin/pretext-tools/issues/12#issuecomment-1368211470

Also, this way we'll have more control over autocomplete. The down side is it will be more work...

Indeed it is a big work to manage those kind of work, and now we start to have robust genereic features like XML completion based on RNG, XML validation based on RNG, code action which fixes problem. We are trying to provide now settings to provide semantic behavior like XML references, XML colors.

I've been playing around and I have a bunch of stuff working. I'll make a PR soon, but here's a screenshot:

Notice the autocompletion for file names :-). The file names are underlined and are links you can click.

vscode-xml could support this feature like XML references, XML colors just with settings like

"xml.files": [
    {
        "pattern": "**/*.ptx",
        "expressions": [
            {
                "xpath": "xsl/text()"
            }
        ]
    }
]

After this configuration, you could benefit with file completion, document link and even validation.

If you are interested with this idea, please create an issue at vscode-xml.

Also, in the lower left there is an "outline" of the document that lists all the builds.

vscode-xml provides the capability to customize the outline with symbols filters (to show only like in your case target). See https://github.com/redhat-developer/vscode-xml/blob/main/docs/Symbols.md#xmlsymbolsfilters

It might be easiest to have vscode-xml handle the xml-oriented tasks and then have a separate language server for pretext-specific things like

goto definition

If you speak about xref, vscode-xml support that.

and refactoring, etc.

Which refactoring kind are you speaking? In vscode-xml we start to provide some code action. See https://github.com/eclipse/lemminx/pull/1429#issuecomment-1381821185

[oscarlevin]

I'm finally finding some time to come back to this, and it really seems like using vscode-xml for our LSP is the way to go. Thanks to @angelozerr for all your help and attention. I'm going to continue to explore new features of vscode-xml as they become available (and have pretext-tools set up the default configurations to use them), but I think we can close this issue for now.