Call To Action: Secure the future maintenance of Xtext

[cdietrich]

Hello Xtext Community,

You might have observed that the number of people actively contributing to Xtext & Xtend has decreased over the recent years. People move on or shift focus. Of course this has implications: Fewer resources mean fewer new features, fewer bugfixes, and reduced community support. On the other hand, the outside world is turning faster: Four Eclipse releases a year, two Java releases, Gradle, Maven, etc. This means that the effort for maintenance and release engineering is increasing at the same time. Building four releases per year including a dozen milestones, adapting to changes in Eclipse Platform, JDT, keeping pace with new Gradle version, changes in new Java Versions etc - all these activities take their toll. Also with the changes in the team structure, fewer and fewer people have the knowledge about the internal workings and the implications of changes inside and outside of Xtext. In a nutshell: the future maintenance of Xtext is a risk.

As there are many people out there using Xtext in their tools and products, this risk has to be mitigated. We are thinking about what can be done to improve the situation. This basically boils down to two things. itemis AG provides a maintenance budget that can be extended by buying support contracts (contact us at xtext at itemis.de for more information or discuss the issue) and/or getting more people involved and contributing. In the past the development and maintenance of Xtext was always a one/two company thing and never a real community effort. So getting interested parties involved in the maintenance process is going to be challenging.

At EclipseCon in Ludwigsburg we talked to a number of interested parties about what we can do and came up with these ideas:

• Make the general maintenance tasks more transparent and document what needs to be done and what has to be done to be part of an Eclipse Release
• Have an open call (Hangout, …) to discuss the roadmap, what needs to be done and who can do what on a regularly basis (4-6 weeks)
• More automation of tasks
• Avoid duplicate tasks (mid-term, repo merge)
• Minimize the risk of things breaking (get rid of old xpand generator/dependency, don’t make extensive use of Xtend with its tight integration into JDT, ASM etc and also no longer encourage users to use Xtend for everything that Java can do as good or even better nowadays, Xtend is no longer the Java 17 of today)
• Think about which features are important and which one can be dropped to reduce the effort
• Think about the release cadence and being part of the Eclipse simultaneous release train.
• Find maintainers for subcomponents like Gradle support.
• Find a better chat-like communication platform than Gitter is atm

This is just a first idea. We invite you to bring in your own ideas into the discussion.

[ewillink]

Sadly you are leading the way on a problem that pervades at least all of the Eclipse Modeling projects. However you should be in a much stronger position so please lead the way. I'm pretty sure that you have many identified rich customers making good use of Xtext. These companies need to be encouraged / shamed into making financial / resource contributions to a support centre.

For me, Gradle is a complete waste of time; I have no idea what it even does. I suspect that it is what Maven ought to be (a not-XML language with 'completion' assist) but I lack the enthusiasm for yet another gratuitous change.

Again for me, Xtend has always been bad news; Java++----. I have consistently recommended writing 'Xtend' as a very thin Xtend layer on top of a thick Java layer. The thin Xtend layer is necessary for M2T String templates. Maybe string templates could reappear in a much simpler form.

[LorenzoBettini]

I think that first of all we should get rid of complex things in the overall infrastructure. For me there are two main points that should be dealt with immediately (of course, for what it can be done in the current bad and sad times):

• Single Git repository! As it used to be in the good ol' days: contributing was so much easier! Now creating an issue might involve provide PRs in several Git repositories (not to mention the temporary adjustment of the branch builds to consume new things proposed in other PRs). You will have to clone the entire Git repository even with things you're not currently interested in, but everything will be much easier.
• Build with Maven and with Maven only. It works! It's rock solid! Just stick with its dictatorial rigid lifecycle structure and it will work! Even Maven support in Eclipse is rock solid! It works! With Gradle in Eclipse having an error free workspace is the exception! Stop building with Gradle (sorry, but I still don't understand all the enthusiasm about Gradle). From what I understand many things break with new versions of Grade... the same does not happen with Maven. Read a Maven POM from any project around and you understand what it does and you can put your hands on it. The same cannot be said about a Gradle build (that's what I've heard at least). Of course, one might still provide gradle plugin support.

The two above points are meant to simplify things! IMHO they should be addressed immediately and should not be postponed.

It is my understanding that performing a release of Xtext is still a painful task. I think that's due to the wrong build infrastructure (too complex). Maybe it might also be due to the JIRO stuff (which I seem to understand did not simplify things... ;)

Concerning the Maven parts I can help. I was suggesting a user BOM some time ago, but then only a dev BOM was created... IMHO, a BOM is for users. I was told that it was too complex (IIRC, because of the complex Gradle/Maven stuff...). I've been working on an Xtext parent POM (currently for my own projects) that drastically simplifies the life of Xtext users and most of all the maintenance of the Maven structure from our side. I can contribute that if there's interest about that.

I already expressed my opinion about Xtend in particular how sad I was that it is not pushed anymore... I hope it's not going to abandoned completely. I'm not sure about that, but probably if also Xbase is abandoned I'll probably stop using Xtext.

[JanKoehnlein]

OK, here's a bit of a TypeFox perspective on this topic.

First, let me state that Xtext is a very stable and mature framework and overall in a very good shape. That's thanks to the maintenance efforts from the committers over the years and recently. But I also think that Xtext wouldn't immediately fall apart if these maintenance efforts were reduced, e.g. by leaving the release train.

The repository split allows each involved party to focus on the individual components they are interested in. And if nobody cares, it's completely OK to abandon a component, as we did for IntelliJ support. And if nobody should care about Xbase anymore, we can still go on with the other components.

For example, xtext-core offers runtime-only or LSP users a small and consistent repo that doesn't scare them away with complexity they don't need. I think that worked pretty well and it has helped to open Xtext to a much wider audience. In the meantime, we at TypeFox almost exclusively focus on xtext-core and language servers. We think it is the future of Xtext.

Being a plain Java repo, xtext-core does not need any Oomph setup, p2 build, target platform or other things that need a lot of time and special expertise for maintenance. Bundling it up again which components that require these would be very counterproductive.

I hope we find a better solution, maybe something with git submodules and a simplified build.

Unfortunately, we don't have a good alternative for Xtend yet. I second that we shouldn't use it for normal code anymore, but for code generation it's still by far the best we have. I tried Kotlin, but like Xtend only works well in Eclipse, Kotlin only really works in IDEA. Maybe somebody knows a reasonably good alternative?

[cdietrich]

yes. but xtext-core is build with gradle. and keeping gradle and the gradle plugin working is work nobody is willing to do

[ewillink]

xtext-core is a complete red herring. There is useful Xtext and there is archive Xtext, In so far as useful Xtext is not a single entity, it is a major PITA.

It seems to me that the unfortunate decision to target IntelliJ caused a lot of bad decisions that have irritated at least some Eclipse users and benefited zero IntelliJ users. GitHub Issues rather than / as well as Bugzilla. Gradle rather than Tycho. GitHub rather than EF hosting ...

Five years ago Xtext was also a modular pain, but one learnt to install MWE2, then a bit of Xpand2 then Xtend then Xtext then MWE2-lang. The composite repos have helped here. Xtext code could be checked out from the one GIT. Since GIT fragmented, I'm just not interested in Xtext's multi-GIT its just too damn hard. Fortunately the good old fashioned Import Plugins and Fragments helped with my last problem.

It would be really good if Xtext went back to one EF GIT, possibly folding in MWE2, Xpand2, Xtend so it is all in the one place. Then users could just checkout the non-archive stuff and it would just auto-build. No need for any OOMPH which never seems to work for my use cases. Reverting to a straightforward Tycho build would solve many other problems.

[JanKoehnlein]

We are not married to Gradle, and we don't plan to maintain the Gradle plug-in either. I propose we migrate the xtext-core build back to Maven. Personally, I'd prefer a plain Maven (non-Tycho) build, because last time I used Tycho (that's a while ago already), it took ages to build because it was scraping a gazilion p2 update sites first and sometimes even failed. If that's no longer the case, I'd be OK with Tycho, too.

TypeFox could do the build migration to keep xtext-core separated. If it makes your life easier, feel free to merge the remaining components and simplify the build. I just wanted to point out that it may be a good moment to decide which of these you want to maintain in the long run before merging them into a giant blob again.

[cdietrich]

i dont think its a good idea to just consider xtext.core and ignoring the rest. as downstream components still depend on xtext-core we need at least an approch that still allows to maintain xtext-eclipse without too many hazards.

[JanKoehnlein]

What are the concrete issues/hazards you see with the separation of xtext-core? Maybe we find solutions that do not involve having to merge all repos.

[cdietrich]

i just want to make sure the eclipse part can be still maintained without having to do extra extra work which would make the situation worse than it is today

[JanKoehnlein]

Please elaborate why a Maven build (which is not the Gradle build nobody seems to be eager to maintain) would make the situation worse than it is today and what your main pain points are with the xtext-core separation. I think I provided a lot of reasons for the xtext-core separation and I am still not clear what you want.

[cdietrich]

i dont know it. i just want to make sure the development works as seamless as it works right now. we know with gradle this work. so we should make sure it works with the new solution too.

if we just have a look at xtext-core and ignore or dont care about the rest this "make sure" is not done

[hannesN]

Here are my thoughts to the current state of Xtext:

There are way to many repositories and build-systems in use right now. I understand that a split between xtext-core and (let's call it) xtext-eclipse might be useful, but 7 repositories come on. If size is a concern I would suggest archiving all git repositories and merge them back to one or two and get rid of the historical files. Nobody needs Xtext 1 or 2.0 in the repository.

I also find it a bit concerning that the maintenance is more concerned with the build process than with the original code. Especially looking from the view of a complete newcomer, Xtext is hell. What we need is a well documented code, generators which are not generating code which throughs warnings or generate comments with URLs which lead to nowhere. This is just embarrassing! I understand the manpower is low. Issues need to be prioritized, but especially the Formatter2 API needs to be finalized.

And here comes the next problem. Because of almost no code is commented or explained, part like the formatter and serializer can't be maintained by anyone but the original programmer. If they leave the team, well we stuck with what we have.

My suggestion: Maintain 2.x but start with a Xtext 3.0. I wouldn't suggest a complete rewrite, however we need to do the following things:

• simplify and document(!) the code, get rid of old APIs and especially get rid of the interface extension pattern and the many instanceof checks if a class implements the newest extension.
• modularize the UI try to get rid of E3 dependencies or at least but them in an additional plugin
• use ONE build system for all modules/bundles and regarding to the eclipse way this should be maven

If the code is easier to understand, the strong dependency to the eclipse IDE is gone, maybe we can find more people interested not only in usage but in contributing to Xtext.

[LorenzoBettini]

We are not married to Gradle, and we don't plan to maintain the Gradle plug-in either. I propose we migrate the xtext-core build back to Maven. Personally, I'd prefer a plain Maven (non-Tycho) build, because last time I used Tycho (that's a while ago already), it took ages to build because it was scraping a gazilion p2 update sites first and sometimes even failed. If that's no longer the case, I'd be OK with Tycho, too.

@JanKoehnlein Tycho is much much better nowadays and it's not the old nightmare anymore. Having a pure Maven build for xtext-core might lead to some additional effort to then turn its JARs into bundles. With that respect, I'd still propose to stick with a Maven/Tycho build for xtext-core itself. It would then be straightforward to create both a p2 site and deploy its bundles as Maven artifacts to Maven central.

[JanKoehnlein]

@LorenzoBettini Thanks, it looks like I am fine with Tycho then.

For all: The main goal of this issue should be to figure out how we can reduce maintenance efforts. So please try to propose new work items only if you're committed to do the work and if that really reduces the maintenance effort in short-term. Please stay constructive by saving your personal frustration and global visions for a better opportunity.

I explained the TypeFox perspective: We want to keep xtext-core as this component is strategically most valuable to us in its current separation. I offered to do the migration of the build which would allow to drop the xtext-gradle-plugin.

Others my have interest in other components. They should please stand up and say what the particular maintenance problems with these are. Then we can try to find a good solution for everybody who's in.

In the end there will be some components nobody wants to maintain. And that's OK. itemis rang the alarm bell loudly (my ears are still ringing) and if no sponsor steps up, we will archive those. As I wrote before, this doesn't mean they are immediately lost. But they will fade away if nobody kicks in. There is no obligation to maintain everything for free forever.

Focussing on the essential components is the only way I see that allows to keep the project maintainable and fit for the future. I am convinced that prematurely merging all repos forces an all-or-nothing situation that will kill the project.

[cdietrich]

just to make it clear: the number resources doing such a thing like: make sure we work with the new ASM 8.0 or 8.1 that pops up on eclipse orbit or work with a new lsp4j version is somewhere between 1 and 2. so we are not talking about developing new features or do bigger bugfixes. it is really about the pure maintenance. exaggerated: if i would not be able to work for lets say 4-6 weeks we would basically have to drop out from simrel immediatly

[ewillink]

Indeed Tycho is very useable, not perfect but one gets used to it. It certainly has many features to accommodate Eclipse. The M2E integration steadily improves. I no longer have fond memories that make me wish to revert to Buckminster.

If you do a full Xtext build, you can spit out an xtext-core sub-repo. In the past this was pretty easy with Buckminster. I never bothered to migrate the functionality to Tycho since the days when memory strapped users wanted a tiny OCL runtime are long gone. They can pull in only a few plugins if that's what they want all by themselves.

[spoenemann]

IMO going back to a monolithic source repository would not solve the original problem (too few contributors for maintenance work), but rather make it worse by scaring away everybody (huge code base, several technologies involved, >1h build times). The real solution is to further decouple the different components of Xtext.

My proposals:

• Decouple the release cycles of lib, core, eclipse etc.
• Drop out of the Eclipse release train.
• Reduce / eliminate use of Xtend in xtext-core (use the generated Java instead).
• Find sponsors for specific tasks (e.g. ASM update) instead of a general "please sponsor Xtext" call.

[ewillink]

What do you mean by build time?

If you are checking Java out of GIT, the auto-build time should be much less than 5 minutes. If not then maybe Xtend is too slow and needs eliminating.

If you mean 'Tycho' build and test, then yes it will take a long time, but a good tool has many tests and that takes time. If it's a real problem you can modularize your tests so that you run only relevant subsets by launching e.g. the xtext-core subset.

[ewillink]

If Xtext drops out of SimRel, it will be difficult for OCL, QVTd and presumably MWE2 and Xcore to stay on the train. If OCL goes then Papyrus, QVTo, ... may be in difficulty.

[tivervac]

What do you mean by build time?

The Xtend compiler is mostly single threaded and very slow, that is where the build times come from.

A lot of work is being done as we speak to reduce the amount of Xtend code.

[LorenzoBettini]

@tivervac if you're talking about Xtend compiler being slow during the build of Xtext base code itself, then the Xtend compiler might be skipped completely during the build since the Xtext Git repositories contain also the generated Java code. It could still be enabled only before releasing since we need the Java trace files for debugging purposes.

[maxweissboeck]

Well, we started a migration from a Visio & Oracle Forms kind of DSL to Xtext mid last year (we are in beta test now) so this could be the kind of message to make us nervous.

So some Questions from me

• Where would be the starting point / information if thinking about contributing to Xtext?
• Do you have any ideas how many active projects with Xtext there are out in the world?
• And why is there no response from the users (like me) to your message so far?

Best regards and thumbs up for Xtext!

[cdietrich]

a first starting point is here: https://github.com/eclipse/xtext/blob/master/CONTRIBUTING.md

as mentioned previously having the code split over 8 repos does not make it easy to contribute so this must be addressed in some form.

one problem with Xtext is that we have no idea how many active projects are out there and how to reach them. thus i dont know how many users of Xtext we I have already reached. Thus i also cannot tell why there are no other user give any signal at all here. But i still think that Xtext is used in a significant number of projects

[miklossy]

Hi Maximilian!

Where would be the starting point / information if thinking about contributing to Xtext?

https://github.com/eclipse/xtext/blob/master/CONTRIBUTING.md

Do you have any ideas how many active projects with Xtext there are out in the world?

https://www.eclipse.org/Xtext/community.html

Hope that helps, Tamás

[cdietrich]

@miklossy this show public projects only.

[miklossy]

@miklossy this show public projects only.

True.

[maxweissboeck]

one problem with Xtext is that we have no idea how many active projects are out there and how to reach them. thus i dont know how many users of Xtext ~we~ I have already reached. Thus i also cannot tell why there are no other user give any signal at all here. But i still think that Xtext is used in a significant number of projects

Regarding to https://www.eclipse.org/forums/index.php/f/27/ current posts are viewed by some 100 people up to some 1000 people

So just speculating, there should be some 1000 Xtext projects outside there, or would people still read posts even without using Xtext actively?

[ewillink]

For the Eclipse Forum, and presumably GitHub, it should be possible for a committer to count the number of subscribed email addresses. Much more reliable than the views count. Once contributing I have probably viewed this thread ten times, while other threads I have been happy to read in Thunderbird and so probably not viewed at all.

[miklossy]

@maxweissboeck @ewillink There are several GitHub user currently watching the Xtext GitHub repositories.

From the Xtext forum as a committer I can see the following statistics forum | number of messages | number of topics | last message

According to the Aggregator Model Editor, at least the following Eclipse SimRel projects depend on Xtext/Xtend:

1. InstallableUnit(org.eclipse.xtext) is required by:

• Contribution(EMF (Core))
• Contribution(EMF Parsley)
• Contribution(EMFT MWE for 2020-06)
• Contribution(GEF)
• Contribution(OCL for 2020-06)
• Contribution(Papyrus)
• Contribution(QVT Declarative for 2020-06)
• Contribution(Sirius for SimRel 2020-03)
• Contribution(VIATRA)
• ...

2. InstallableUnit(org.eclipse.xtend.lib) is required by:

• Contribution(ECP)
• Contribution(Lsp4j)
• ...

[OLibutzki]

As I use Xtext from almost the beginning I'd like to share my insights.

I still use Xtext on a regular base, because I do not know a comparable Java framework. It was a good design decision to separate the language runtime and the ui stuff. The language runtime is independent of Eclipse which is crucial for the future of Xtext imho.

The Eclipse integration of Xtext is awesome. I like the APIs and I like how straightforward it is to adjust the behaviour although the defaults are already quite smart.

Other IDEs are getting more and more traction and the need for editing DSLs within web applications becomes more important/essential.

The Xtext documentation should reflect this shift. In the docs the "old" Web editor support is documented, but I do not find a word about LSP.

How can I use Xtext DSL in VSCode, Theia, IntelliJ(?), my own web application? In my opinion this stuff needs at least as much attention as the Eclipse chapters.

The runtime which is "Plain Java", the Eclipse integration and the IDE agnostic LSP are Xtex's future and the LSP docs need to catch up.

---

One comment regarding contributions: Some years ago I tried to contribute a very small bugfix or enhancement (can't remember anymore). It was a change of about 10 characters. For hours I tried to set up the IDE and tried to build Xtext for this tiny contribution. It was a mixture of Gradle, Maven, Tycho and I felt lost. In the end I just created the pull request without having an IDE or a build tool which built my change.

Maybe the IDE setup und build infrastructure have improved in the meantime, but I just would like to provide the feedback that this complicated setup deterred me.

[maxweissboeck]

I just can second this, Xtext is amazing! And yes, a simple Web Example, like the classic examples, would be really great and may get a lot of attention!

[szarnekow]

I have been chewing on this topic for quite some time now and was reluctant to jump into the discussion so far to avoid an impulsive response.

Now I'll try to focus on "Secure the future maintenance of Xtext". Discussing all the technical dept (which I am and the team actively working on Xtext is well aware of), lack of (web-)examples, potential for improved documentation (balancing the feedback between "documentation is too verbose" vs "document is missing" turned out to be an almost impossible twist in the past) and features that could be added - all this is out of scope for this ticket and discussion here I suppose. Of course these are important parts that will or will not contribute to the success of the framework, but as Jan already mentioned: Putting more work on the table won't help much with the problem at hand.

Trying to figure out why we do have this situation where the main contributing party is ringing the alarm bells, I've a hard time to pinpoint a single reason how this project ended up here. For sure, there were attempts to make Xtext more attractive for contributors. The most obvious technical attempt to do so was the repository split. Given the numbers of contributions since then compared to the status with a mono-repo, I think it's fair to say that this was a failed attempt. For me personally, the large number of repos is just making every single action unnecessary hard and annoying. It is almost impossible to make a change anywhere in the codebase without running all downstream tests which in turn requires changes in all repositories. Not fun at all. The unnecessary complex build on top of that doesn't simplify things either. Obviously the contrary is true, as Oliver pointed out. The need for multi-step shell-scrips even on the most basic of project in our potpourri of repositories is already a clear indication for something that went utterly wrong.

To me, it looks like an attempt to attract new contributors is prone to fail at the moment since most of the tasks that are to be done are neither sexy nor intellectually challenging. Therefore I think it is most important to reduce the burden on those few people, that are currently doing the work. Only if we can make it easier to keep everything(tm) consistent and increase the automation towards one-click solutions for the maintenance tasks, we'll manage to find the time again to work on future topics - which include mentoring of contributors.

To propose concrete action items:

• Be more transparent with the ideas about the future of Xtext - last year at EclispeCon a (bi-)monthly hangout was discussed, maybe that's worthwhile to have
• Continue to remove Xtend from the Xtext codebase to speed up the local build and become ready for future Java versions
• Merge the repos (Contrary to Jan and Miro, I don't think xtext-core alone is a reasonable subset given that editing grammars is only well supported in Eclipse as far as I can tell)
• Use Tycho as the single build technology that is capable of building plugins
• Enable API tooling in the build to spot regressions early

I don't want to appear overly optimistic: None of these measures will help with securing the future of the framework beyond a state of 'Heartbeat present'. Either interested parties step up and do "something" or ... well - the alternative is probably obvious.

[ewillink]

There are two separate futures that should not be confused.

A better Xtext will no doubt have new enthusiastic committers who can do something useful.

A stable (life support) Xtext will continue to offer existing functionality accommodating the incessant evolution of Java/platform/Tycho/Jenkins/...

If necessary, I might step up to keep stable Xtext builds ticking over as I have done for the MoDisco project. But a pre-requisite would be a simple buildable Xtext; one repo, boring Tycho, ... As it stands the repo split marked the day that I could no longer attempt to understand the project.

[drkstr101]

Greetings all,

While I am still too new to Xtext to offer any meaningful support, I came here to answer the call by offering to update/maintain documentation, create code samples, test cases, etc. as needed. I could also possibly help keep Gradle support around, as I have a bit of experience with this already.

As I read through the comments here, I thought it may also be beneficial to provide a perspective from the next generation engineers, whom you will need to answer this call for the benefit and longevity of the community.

Other than a vague understanding of what it is, I was just exposed to Xtext for the very first time about a month or two ago. I have always noticed that Eclipse tooling has a special shine to it that is lacking in many (all?) of the other open source alternatives. I never knew why this was, but hello, Xtext!

Xtext opened up whole new and amazing ways of doing things to me. I was immediately star-struck, with visions in my head of all the amazing possibilities.

This is not specific to Xtext itself, but building Eclipse projects have been such a foreign experience to me, that if I wasn't so damned stubborn and persistent, I probably would have given up long ago, as I believe some like I have done.

I believe these two things will greatly reduce the barrier of entry for developers like me to contribute or even use Xtext in my non-eclipse projects.

1. Phase-out Xtend in favor of testing against Groovy/Kotlin/Scala/JavaX support

2. Further encapsulate components so they can easily be understood, tested, and maintained on their own. Provide copious amounts of documentation and working example snippets, which gets validated during CI (EG. living documentation).

I myself will be happy to contribute to the best of my abilities, as soon as I have mastered enough knowledge to do so. Until that day, the best I can offer is docs and example/test cases.

Best Regards, ~Aaron

[obigalk]

Hi Folks,

I would like to contribute to Xtext. I have some very fancy ideas to improve the tool set of Xtext.

I got to know Xtext about 15 years ago and was enthusiastic about what Xtext could do for me.

If you have written a grammar in Xtext the tool set generates a number of stubs to implement different aspects of surrounding tools for code generation with Xtext. The most important ones are ProposalProvider, OulineProvider, LabelProvider, Validation all the aspects important for the user who should write instances of your Xtext grammar. I very fast understood that Xtext could have generate much more flesh into the stubs than it currently does.

Over years I developed my own tools to implement these things by its own special Xtext grammar. Unfortunately this tool set got out of date due to the changes Xtext had evolved.

I recognized that Xtext meanwhile supports annotations in the grammar which makes me hoping that some of the mentioned tools can be generated from the grammar annotations. Some of the tools could be generated by elaborating or replacing the stub generators in Xtext. Unfortunately this is only possible inside the implementation of Xtext since most of the classes that need to be changed/replaced are not accessible from outside.

If there is interest in knowing more about my ideas, please answer or contact me.

Best Regards ~Olaf

[ewillink]

Your thoughts echo many of mine. Xtext made huge strides in its early years. A pity these have not been followed up. Why don't OOMPH, Sirius, Xtext use Xtext DSLs? Formatting. This can certainly be handled by some judicious annotations that match standard idioms. But you will make the *.xtext file unreadable. My proposed solution to this is a multi-view editor. The basic EBNF view shows only the grammar layer, no CS details. The Annotated EBNF view corresponds to Xtext today; grammar plus CS annotation layers. Differently, the formatting view shows grammar plus formatting annotation layers with helpful completion assists and maybe even policy wizards. You can add more views for more concerns. Outline. This is very idiomatic and poorly handled by the declarative style. There are too many choices and no help for annotations . An outline (Xtext) DSL would be much much better, readable with useful content assist/validation. The outline is currently recomputed because incremental update is too hard. Given a sensible DSL, a generator could emit code for an incremental engine such as EMF-IncQuery / QVTr. (As an experiment towards this I produced a modelled outline using OCLinEcore. See https://git.eclipse.org/r/plugins/gitiles/ocl/org.eclipse.ocl/+/master/examples/org.eclipse.ocl.examples.build/model/UML2EcoreControl.ecore where it is surprising how many helper methods are useful.)

[dunnry]

My company is a pretty extensive user of core Xtext for last couple years. Our focus is solely on Xtext core and LSP support as we see web-based editors as our future. From our perspective, having separate, focused repositories is a benefit. I was able to get xtext-core repository cloned and working locally relatively smoothly and even submit a PR. I suspect that my company will contribute back bug fixes as time goes on and keeping the repositories/issues/PRs focused to what we are using will make that easier for us to follow, adopt, and contribute back.

It would be nice to decouple grammar creation (.xtext and .xcore) from Eclipse IDE and move to LSP compatible editors (stretch goal). I agree that it would also be nice to phase out Xtend from core usage. We are finding it very difficult/impossible to debug Xtend code outside of Eclipse (e.g. from IntelliJ). That will also make it easier to find/fix/contribute in core project.

[AndySydow]

I used in several projects (at least one quite big one) Xtext and I'm always impressed about the possibilities. But such perceptions

Also with the changes in the team structure, fewer and fewer people have the knowledge about the internal workings and the implications of changes inside and outside of Xtext.

are sad. I can't provide fast ideas how this can improve, but I think budget/support contracts alone won't fix it (esp. based on the quote). As a guy from "outside" I can give one feedback, that it is quite hard to get into it to support development/maintenance. I shortly checked Issues with "good first issue" tag ... sorry but mostly there is no detailed description and/or eclipse bugzilla link with also no further description.

I'm self-employed, so I have the ressource to support. But with missing documentation and no established way to share knowledge (or at least not clear to me) its hard.

[ewillink]

@Andy. Yes documentation is usual scant, but at least it's not then stale.

If you are seriously interested in helping, you will find that there are developers/users who will endeavour to help mentor you. (And of course you can always improve the documentation.)

[drkstr101]

To anyone who would like to contribute but is worried they may not have the skills to do so, I have found the Xtext community to be very warm and inviting. I came into this with practically zero knowledge of Xtext beyond how its used, and they were able to get me up and running in no time. There are lot of things that can be done that require no specialized knowledge, and will begin to expose one to the concepts behind Xtext.

I encourage anyone who would like to contribute to speak up and join the fun!

[ktk]

Some asked for more feedback by users so let me chime in. I know about XText for maybe two years, one of my colleagues proposed it to create a DSL in the domain of RDF Knowledge Graphs and I am very happy he convinced us to try it.

Fast forward we finally released our work to the public last week, after using it for a long time in internal projects. I love what I can do with it and we have many ideas of other applications for XText based DSLs in our domain of work.

The plugin works great in Eclipse, refactoring mappings is a joy and it just works flawlessly. We also release a version for Visual Studio Code using XText LSP. It works but not nearly as good/complete as the Eclipse plugin.

We compiled a list of issues of things that work in Eclipse but do not work in the LSP version and we planned to approach one of the companies offering professional XText support to see how they could be addressed (we could share that list btw). I would love to have more money by customers and push that into XText as well but unfortunately, we are not there yet. People want DSLs with refactoring support but few want to pay for it, it seems.

From our perspective, LSP is vital, in my opinion, Eclipse Theia or Github Codespaces are the future for many enterprise-environments. Also, many do not want to or cannot install Eclipse on their internal machines.

So yes, more than happy to work on a path where we can contribute financially as well. We are very interested in decoupling XText more from the Eclipse IDE and make the LSP version as powerful. I'm also convinced that this would open a whole new group of potential customers for XText based work.

Thanks to everyone for the work!

[ewillink]

Formatting. .... My proposed solution to this is a multi-view editor.

I had a much better idea; a truly declarative formatter based on a formatting idiom abstraction. I will work on this as https://bugs.eclipse.org/bugs/show_bug.cgi?id=564265 initially for use by OCL and QVTd editors, but if it works perhaps an Xext new-new formatter.

[MarkusAmshove]

As a long time Xtext user for multiple in-house DSLs I thought I should also share my opinion on this topic.

Regarding the the ease of contribution discussion, I’ve tried to contribute several times in the first few years of using Xtext, but I’ve never really gotten anything to work, because it felt like a huge step. The problems were about Bugzilla, some kind of Eclipse Commiter thing that cloned some repositories and all of the sudden I’ve had multiple projects that I couldn’t get to build etc. so I stopped there every time.

The first welcoming change was moving to GitHub for issues. This is a place I already know and don’t need an additional account for, that was great!

The next thing is the release of LSP as a target for Xtext and therefore me being able to use Gradle or Maven in my projects instead of Tycho. Using P2 behind a corporate proxy is the most fragile, least amusing build system I’ve ever had to work with, even with Artifactory as a proxy. Our Xtext projects that are still producing eclipse plugins are stuck at old Xtext releases because we’re scared to change versions and break Tycho builds. The same goes for the target Eclipse instance. With the introduction of “normal” maven repository stuff (aka not using P2) for Xtext Core and the LSP I’ve never had any issue with building a project or updating the Xtext version on projects that don’t create Eclipse plugin (I just have to increase the version in the build file).

I have to admit that I haven’t tried to contribute changes now that the repositories are split and issues hosted on GitHub, but I would at least give it at try again on non-Tycho-repositories (Core and LSP I think), so the entry barrier for me as a possible contributor (just a things person though) is higher with the more modular approach.

The change from Xtend to Java for core and LSP would also open up a greater variety of possible contributors for those two, namely people not using Eclipse and therefore not being able to use Xtend.

I could also help on the Gradle side, as we’re using Gradle instead of Maven (but I have no problem using maven, as long as it is not Tycho 🙃), as I’m responsible for the build stuff in our company and I’ve experience in developing gradle plugins.

[mvmn]

IMO:

1. First and foremost, you have to make people care about Xtext. It should be promoted more, and it's benefits should be more evident to wider audience. This, however, may include some shift in focus - Xtext should no longer be "the Eclipse thing" that is "made to build DSL editors in Eclipse". Very few people/companies care about that. Web editor is far more important. Perhaps LSP as well, though it's way more complicated for no apparent reason (who actually builds their prod APIs as json-RPC? Nobody - but LSP).
2. Second, all "custom" stuff should be reduced to a minimum (ideally removed). Why average project can be easily build with Maven or Gradle, while Xtext requires MWE2, Tycho and that sort of nonsense? Clearly there is a historical factor here - Xtext was made for Eclipse. But, again, IMO this has to change. Otherwise Xtext will just fall out of relevance. Xtext should be a pluggable thing that would be easy to plug into any IDE, web editor, LSP server, you name it. Same goes for Xtend - it should not be there! The code gen should be pluggable, so anyone could write their own plugin and generate with what they want. Xtend is so niche it's impossible to understand the need for it's existence at all. Kotlin support is an issue in Eclipse? How about Groovy then for default codegen plugin implementation? It's not idea, but again, it should be replaceable with any custom plugin for anything people want.
3. In order to achieve any of this there is a need for a ton of contributors who understand Xtext. For this there is a need to create facilities for people to be able to understand architecture of Xtext more easily, and things it's built upon.

[ewillink]

It seems clear that we have two communities.

a) Eclipse Xtext users hate all the migration away from traditional Eclipse b) Universal Xtext users hate the use of traditional Eclipse.

If there was a large support community, the migration to universal that Xtext attempted five years ago might be progressed so that it actually worked and supported Eclipse as just one of many target platforms.

However the migration to universal Xtext foundered and the support has retrenched - no more IDEA. Further retrenching to traditional Eclipse is comparatively easy and can be accommodated with modest support. If there is a genuine community for a universal Xtext, let it demonstrate that it has the resources to create something good rather than yet another variant of poor.

[alexandrebouchard]

First of all, Xtext is really awesome, thank you for sharing this amazing project to the world and fingers crossed it finds a second wind and continues to evolve.

Just wanted to bring three arguments in defence of Xtend.

First one is its suitability for scientific/math oriented languages: we have built an open source probabilistic programming language using Xtext (https://www.stat.ubc.ca/~bouchard/blang/, preprint available at https://arxiv.org/pdf/1912.10396.pdf) and several features in Xtend were instrumental, in particular features which are not coming anytime soon in Java. E.g.:

• operator overloading: critical for things like matrix algebra and auto-diff
• extensions and dispatch, which are quite handy for the rich type zoos that arise in building mathematical libraries

These alone actually make Xtend superior by a large margin to Java for any scientific applications in my opinion. I speculate that might be a non-trivial user segment.

Second thing that differentiates Xtend to Java and IMHO could guide its growth is that language developers can assume Xtend will have good IDE tooling available which is not the case for Java. So this means certain language features, which would never be good idea for Java, can be quite productive in Xtend. Consider for example aggressive type omission, complex scope rules, implicit it, small-talk style lambda arguments, etc. In Java, these would alienate non-IDE user, so they will never come to be. In contrast, these advanced features work very well in Xtend thanks to its guaranteed tooling. I suspect there is more creative language design that could be done on Xtend based on the high quality IDE tooling.

Third is dog-fooding: it ensures Xtext development uses Xtext, increase chance of discovering opportunities for improvements, etc.

In summary I think Xtend is great. I am writing quite a bit of research code in Xtend right now and it is an outstanding fit as a language for that application in my opinion.

[gerdleon]

Formatting. .... My proposed solution to this is a multi-view editor.

I had a much better idea; a truly declarative formatter based on a formatting idiom abstraction. I will work on this as https://bugs.eclipse.org/bugs/show_bug.cgi?id=564265 initially for use by OCL and QVTd editors, but if it works perhaps an Xext new-new formatter.

We also wanted to improve on the formatter and the result is this https://ddk.tools.avaloq.com/format_quick_start.html

[EricSalemi]

I think I belong to the category of person who would love to see XText become a universal tool.

I come from the software industry. In my surroundings nobody uses Eclipse. I have now worked in three separate companies over 12 years and they all made the move to Jetbrains tools. As a devops architect I find myself developing DSL quite often but the support of XText in IDEA is so poor (Removed native code, LSP documentation inexistent) that I have to use other tools. It is a shame because I like the idea of Xtext but it is not possible to convince any manager with a situation like this. My sincere opinion is that a large community will never form around XText until it proves it really wants to get Eclipse away from its core. I understand where XText comes from and how Eclipse users would find this move unpleasant, but the vision of XText cannot please everyone. What kind of community XText wants? The question remains unclear to me.

On Tue, 30 Jun 2020 at 23:18, gerdleon notifications@github.com wrote:

Formatting. .... My proposed solution to this is a multi-view editor.

I had a much better idea; a truly declarative formatter based on a formatting idiom abstraction. I will work on this as https://bugs.eclipse.org/bugs/show_bug.cgi?id=564265 initially for use by OCL and QVTd editors, but if it works perhaps an Xext new-new formatter.

We also wanted to improve on the formatter and the result is this https://ddk.tools.avaloq.com/format_quick_start.html

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/xtext/issues/1721#issuecomment-652049379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFKIEVKBKUARJXLW673LRQLRZJJCVANCNFSM4LWQOK7Q .

[ewillink]

Your comments epitomize the problem. Software industry that happily exploits Eclipse projects without contributing anything back. Perhaps in the early days of Eclipse when IBM was a massive contributor this might have seemed sensible; contributing into IBM was not possible. Now it is the cause of many Eclipse projects demise. If software industry wants 'free' software, it must contribute to the salaries/pensions/health schemes of the maintainers. If you want a Universal Xtext, then you must combine with other like-minded companies to provide the funding. Else you must stop complaining when the existing minimally/unpaid maintainers retrench to something simpler.

[mvmn]

Software industry happily exploits Eclipse by not using it but using IntelliJ IDEA instead? Right...