Open tuxji opened 2 years ago
dhuebner
commented
2 years ago I think it is because of changed dependency to lsp4j 0.12.0, which now uses lsp 3.16. So you can either switch back to Xtext 2.25 or will need to migrate to the new Xtext 2.26 API
crapo
commented
2 years ago @dhuebner , I'm having a hard time finding what has changed in the "new Xtext 2.26 API" that is relevant to this problem. I'm not finding anything about how to replace ISemanticHighlightingStyleToTokenMapper. Can you point me to something to help me identify what has changed and how to use the new API?
dhuebner
commented
2 years ago @crapo
I think ISemanticHighlightingStyleToTokenMapper was just removed, I also need to reimplement semantic highlighting in an other project https://github.com/theia-ide/yang-lsp/issues/221, but still not looked into it.
I would probably start looking into semanticTokens changes made here https://github.com/eclipse/lsp4j/pull/446
dhuebner
commented
2 years ago @crapo I pushed a first working draft version for Xtext 2.26 + LSP4j 0.12.0 (LSP 3.16). Maybe it can help you to start in SADL https://github.com/theia-ide/yang-lsp/commit/4428d196077402ec40f7ff43cd0c4c24d7629bb3
crapo
commented
2 years ago @dhuebner . this is very helpful. Thanks for sharing!!!
baoluomeng
commented
1 year ago Hi Andy @crapo, I am Paul Meng from GE Research. I am now part of Kareem's team. John Interrante @tuxji and I are working on integrating SADL/Eclipse with OSATE and came across this issue. The problem is that OSATE uses xText 2.27 and SADL is using xText 2.25.0, which causes the incompatible issue. So I wonder if you are planning to resolve this anytme soon?
crapo
commented
1 year ago Hi Paul @baoluomeng , as you can see from the comments on this issue, I started looking into it when @tuxji reported it last March. However, I was unable at that time to get a clear idea of what had changed and how to accomplish the SADL code changes to go to a new API with confidence. Over much of the life of SADL, we have brought in consultants who were Xtext experts to aid in making the most of Xtext's capabilities. Initially we used itemis and once Sven Efftinge left itemis and started TypeFox, we used TypeFox. This had enormous benefits--it allowed us to focus on the semantic modeling aspects while getting the best out of Xtext at a much lower cost than if we had tried to leverage Xtext on our own. I would recommend the same approach to this problem. I'd be happy to work with an Xtext expert consultant to make sure that the modeling capability of SADL is preserved and maybe enhanced and that the modifications necessary to go to the new semantic highlighting API are robust. Akos Kitta at TypeFox has deep knowledge of how Xtext is used in SADL. I don't know if there is any chance of contracting for some of his time, but I believe it would be by far the best approach. Please let me know if this is a possible direction.
spoenemann
commented
1 year ago Hi Andy and Paul! I'd like to inform you about changes at TypeFox in the recent years. Sven and Akos are no longer at TypeFox – a part of the team moved to the new company Gitpod when it was founded in 2020. I am now leading TypeFox together with @danieldietrich; see also this blog post.
We still have Xtext experts, though, @dhuebner being one of them. So continuing support for SADL is generally possible. But I'd also like to highlight an alternative: Langium (langium.org). We started that project in 2021 as a successor of Xtext in order to get back to a high development speed and become compatible with the web platform. The main difference is that Langium is fully built on TypeScript; it can run both in desktop IDEs / backends (Node.js) and web apps / frontends.
If you'd like to discuss more in depth: miro.spoenemann@typefox.io
crapo
commented
1 year ago I took another look at this issue by downloading the latest Eclipse for DSL, 2022-12 and running the Xtext examples. It doesn't appear that any of them implement semantic highlighting or any explicit highlighting functionality. Nor do the tutorials give examples. The example that @dhuebner referenced above, see https://github.com/TypeFox/yang-lsp/commit/4428d196077402ec40f7ff43cd0c4c24d7629bb3, is certainly helpful, but doesn't give the same level of understanding and confidence that a good tutorial might provide. The information at https://www.eclipse.org/Xtext/documentation/340_lsp_support.html has a link for Semantic Tokens (aka Semantic Highlighting), but once again the high-level discussion of how semantic highlighting is now implemented, and how that relates to the prior implementation, is, as nearly as I can tell, entirely missing. @spoenemann , as far as langium use is concerned, I can imagine that this would be a better technology and make Web SADL and Eclipse SADL more compatible, but I can only imagine that moving SADL and its extension languages and IDEs to an entirely new platform, with a different parser, would be a major undertaking. The architecture of SADL was modified by @kittaakos to support both Web SADL and Eclipse SADL, but at the beginning of efforts to support both in the same Xtext-based implementation. I suspect this work makes the modification to support the new semantic highlighting API a little more complex. @baoluomeng , I assume that you are following this conversation...
baoluomeng
commented
1 year ago I was traveling. Yes Andy, I am following this conversation. @crapo This does not seems like an easy-to-solve problem. I will need to talk with our team to see how to proceed. In the meantime, we can work with the old Eclipse.
tuxji
commented
1 year ago I have updated SADL to Tycho 3.0.4, Java 17, and Eclipse 2022-12 in PR #1063. Please test the latest build artifacts and let me know if any important SADL functionality has been affected or not.
crapo
commented
1 year ago I'm curious to know if anyone has a long-term plan for semantic highlighting in SADL. SADL without semantic highlighting seems to me to be much less appealing. The Xtext move to LSP and the introduction of the replacement platform Langium poses significant challenges. The long-term support of Xtext is in question as documented in this Xtext issue: https://github.com/eclipse/xtext/issues/1721. I do see that some tools for moving applications from Xtext to Langium have been developed since this issue was first created: see https://www.typefox.io/blog/xtext-to-langium. @tuxji , @baoluomeng , can you share information about the current level of usage of SADL and commitment to maintaining a semantic highlighting capability in SADL?
tuxji
commented
1 year ago I don't want to spend any more time than necessary updating SADL to work with the latest Java, Xtext, and Eclipse. SADL still does some syntactic coloring (highlighting) of the text when you edit a SADL file in Eclipse. To me, the SADL file still seems to look the same in Eclipse as it did before. I am hoping that whatever additional functionality semantic highlighting provides (I'm not familiar with what semantic highlighting does) will not be missed that much, but if we get feedback and comments from the RACK development team that SADL is much less appealing without the semantic highlighting, I can try to emulate the same two commits that Dennis (@dhuebner) made to TypeFox/yang-lsp. Let's try the current changes and get some feedback from people who currently use SADL like the RACK development team.
crapo
commented
1 year ago With semantic highlighting (e.g., prior to your latest removals, a simple model looks like this:
Using the build artifacts you reference above, the same model looks like this;
In other words, semantic highlighting is what gives color coding to classes, properties of various types, instances, variables, and a few other things.
tuxji
commented
1 year ago Yes, I now see the effect of losing the semantic highlighting functionality and the loss of the color coding does seem significant. Thanks for showing me the difference.
crapo
commented
10 months ago @tuxji , I don't know if you've gotten around to thinking about semantic highlighting re-implementation in SADL, but I've been doing a little reading on language servers and want to just highlight (no pun intended) a couple of things.
What are you thinking is the best approach? I'm thinking that we should consider implementing a language server for SADL in Java and it can then be used by both Eclipse-based SADL and WebSADL, which can be updated if desired to use the latest version of VSCode.
tuxji
commented
10 months ago Yes, I think implementing a SADL language server is the way to go like @dhuebner did for his project. I'm not sure if SADL already has a language server or if we need to create one from scratch. I see some signs that there is a language server, but I don't see all the files that @dhuebner used.
crapo
commented
10 months ago There is a language server I believe, used by WebSADL, implemented by Akos, and in Typescript I think. That was before LSP supported semantic highlighting. That's why I think maybe re-implementing in Java might be the way to go.
dhuebner
commented
10 months ago @tuxji What parts are you missing? Btw. you can use the "New Xtext Project" wizard to see how to create a LS for an Xtext project.
crapo
commented
7 months ago After repeated failed attempts, I have finally successfully implemented the example in the Xtext LSP-Support document. I submitted a PR on that document to fill in a little of the missing information. This demonstrates using Eclipse LSP4J and LSP4E. I think that the existing SADL code base has everything necessary for using LSP for code completion, semantic highlighting, quick fix, folding, etc. It's a question of refactoring everything to access the correct pieces of implemented capability through the language server. There is also the question of the client. The LSP-Support example uses the Eclipse Generic Editor. Presumably we could continue to use an Eclipse Xtext editor. Or we could use Theia for both the fat client and the Web client. As I understand it, if one did that there is little effort to support VSCode as the client as well. @dhuebner , @tuxji , does this all sound accurate? @tuxji , have you done any work in this area since we last discussed it?
tuxji
commented
7 months ago Hi @crapo , I have kept some browser tabs around ready for when I come back to this area, but I have not had time to do any work with LSP support in SADL yet. However, I will have time in December to help add LSP support to SADL, so let's start working together on that effort. SADL uses the Xtext editor for editing SADL files, so let's do just enough refactoring and programming to support LSP using the Xtext editor. Let's not concern ourselves with Theia and VS Code right now.
crapo
commented
6 months ago Hi @tuxji . I have been trying to get my head around Language Servers in Eclipse in general and Xtext in specific. Among other things, I worked on the example at https://eclipse.dev/Xtext/documentation/340_lsp_support.html. This is missing a critical piece of information--what to do with the TextMate json content. I finally figured out that one sames it as a file (...tm.json) and then adds that to the TextMate Grammar in Preferences. I submitted a PR which was approved but hasn't been incorporated. You can see the relevant screenshot here. I finally got this example working as expected and have then tried to expand upon it to have a custom LS instead of the generic one, but so far haven't quite got that to work. I guess I don't understand how the binding works so that I can bind the custom LS so it gets used.
This link also seems to be very relevant. There are lots of pieces already in the SADL code base. In particular, in the ...sadl.ide project:
I've looked at lots of other stuff, but haven't found anything else that seems directly useful.
Please let me know if the solution to any of these problems is obvious to youi.
tuxji
commented
6 months ago I googled "Unknown extension point" and someone said to double check that the plugin(s) defining the extension point are dependencies in your plugin.xml's accompanying MANIFEST.MF. I ran "git pull" in my sadl checkout hoping to see your changes in a branch so I could troubleshoot further, but you haven't pushed them to the repo yet. My only other advice (besides pushing your changes) is to look at some of the projects mentioned in tm4e's README.md as using tm4e in case one can serve as a good role model/example to emulate. Also, https://idiomaticsoft.com/post/2022-04-25-lsp-xtext-tutorial-intro/ may be worth a look.
crapo
commented
5 months ago @tuxji , let's pursue this first with a minimal example. I've tried to move the result of the example at https://eclipse.dev/Xtext/documentation/340_lsp_support.html to a plug-in environment but have some issues. I've pushed this minimal example to LS-Exploration branch as a set of nested projects parallel to sadl3 in the github repo. This set of project builds with cmd-line Maven without error. However, I can't get the Eclipse environment to resolve LSP4J references. Possibly related, I don't understand why I don't have any Maven Dependencies listed in the Project Explorer. The quick fix on the unresolved references is to add to the target platform, but when I pick that and then pick LSP4E to add, nothing happens.
If you can help me get these issue sorted out, then we can work on follow-on issues:
Then I can work on customizing the LS. This will give us a working example to guide the implementation in SADL.
tuxji
commented
5 months ago @crapo , I read https://eclipse.dev/Xtext/documentation/340_lsp_support.html and I do not see a link to download source code; rather, I see instructions to write source files with some copy and paste. Is that how you created the source files in the new org.example.domainmodel.parent directory, by following these steps manually in Eclipse? In particular, I want to know where the Maven pom files came from (did Eclipse generate them when you created the projects)? Ultimately, I'm looking to either compare the source files in org.example.domainmodel.parent against a set of original source files or follow the same steps you followed while I make improvements along the way.
crapo
commented
5 months ago @tuxji , the instructions in 340_lsp_support.html do not create Eclipse plug-ins but rather just add an Xtext grammar and the default LS to the current Eclipse environment. (See the first screenshot in that documentation.) The instructions send you off to another tutorial to follow to get the contents of the grammar. I'm not aware of any source available for the project. The pom files are created by the new Xtext project wizard I guess. Following these instructions, I did get everything working in the original Eclipse environment. In other words, you create a .dmodel file right in the development eclipse, you don't run a new Eclipse with developed plug-ins.
However, for our minimal example we want the new Xtext project wizard to create plug-ins so I followed these directions as closely as I could but chose "Eclipse plug-in" in the Advanced Xtext Configuration dialog shown in the first screenshot. I then copied over the several classes from the working project created when I followed the directions exactly, and some classes that I created to provide a customized LanguageServer, namely:
As well as the TextMate Grammar file:
The poms were, once again, created by the new project wizard, Maven being chosen as the preferred build system.
I also added LSP4E by through the Help -> Install New Software..., using https://download.eclipse.org/lsp4e/releases/latest/ as the installation site. I checked the first three options but also tried with all four options checked.
Note: I found and fixed some typos in names in writing this comment, so I suggest pulling changes from github in order to keep things consistent.
crapo
commented
5 months ago @tuxji , just to confirm that we're on the same page, I'm hoping that you can help get the architecture understood and working. Once that is in place I should be able to use my knowledge of the codebase for semantic highlighting, content assist, renaming, etc, to get all necessary functionality moved over to the LS.
tuxji
commented
5 months ago OK, I'll make some time next week to help get a LS project working.
Xtext 2.26.0 appears to have changed or removed some highlighting API used by SADL. When SADL is installed in Eclipse 2021-12 with Xtext 2.25.0, it can open and highlight a SADL file. When SADL is installed in Eclipse 2022-03 with Xtext 2.26.0, it encounters exceptions when trying to open and highlight a SADL file.