eclipse / xtext

Eclipse Xtext™ is a language development framework
http://www.eclipse.org/Xtext
Eclipse Public License 2.0
766 stars 319 forks source link

Call To Action: Secure the future maintenance of Xtext #1721

Open cdietrich opened 4 years ago

cdietrich commented 4 years ago

Hello Xtext Community,

You might have observed that the number of people actively contributing to Xtext & Xtend has decreased over the recent years. People move on or shift focus. Of course this has implications: Fewer resources mean fewer new features, fewer bugfixes, and reduced community support. On the other hand, the outside world is turning faster: Four Eclipse releases a year, two Java releases, Gradle, Maven, etc. This means that the effort for maintenance and release engineering is increasing at the same time. Building four releases per year including a dozen milestones, adapting to changes in Eclipse Platform, JDT, keeping pace with new Gradle version, changes in new Java Versions etc - all these activities take their toll. Also with the changes in the team structure, fewer and fewer people have the knowledge about the internal workings and the implications of changes inside and outside of Xtext. In a nutshell: the future maintenance of Xtext is a risk.

As there are many people out there using Xtext in their tools and products, this risk has to be mitigated. We are thinking about what can be done to improve the situation. This basically boils down to two things. itemis AG provides a maintenance budget that can be extended by buying support contracts (contact us at xtext at itemis.de for more information or discuss the issue) and/or getting more people involved and contributing. In the past the development and maintenance of Xtext was always a one/two company thing and never a real community effort. So getting interested parties involved in the maintenance process is going to be challenging.

At EclipseCon in Ludwigsburg we talked to a number of interested parties about what we can do and came up with these ideas:

This is just a first idea. We invite you to bring in your own ideas into the discussion.

rasifix commented 3 years ago

If you want to scare contributors away, go with Tycho. IMHO, Tycho builds are generally a mess that is hard to understand and maintain. It is a necessary evil for Eclipse plug-ins, but please don't make it mandatory part for xtext-core and LSP. I also see xtext-core and LSP as more of the future for Xtext than the Eclipse integration (which is cool too, but limited to a smaller and smaller percentage of the user base).

rasifix commented 3 years ago

Reverting to a straightforward Tycho build would solve many other problems.

and introduce a lot of PITA for all

cdietrich commented 3 years ago

the main problem is: there is more people with tycho knowledge than with gradle knowledge there is a reason we are still at gradle 5.6

rasifix commented 3 years ago

the main problem is: there is more people with tycho knowledge than with gradle knowledge there is a reason we are still at gradle 5.6

you are trying to attract more potential contributors. IMHO, Tycho is a major detriment to achieve that.

cdietrich commented 3 years ago

well why there nobody contributing a gradle 6 update then?

rasifix commented 3 years ago

well why there nobody contributing a gradle 6 update then?

Maybe because updating a build system is associated with lots of pain and not really "cool". People would rather prefer doing real substantial stuff instead.

I don't say it has to be Gradle. Build for xtext-core and LSP could be plain Maven (without Tycho).

hannesN commented 3 years ago

Sorry but I doubt it, that Tycho is the Showstopper. There is no need to modify the poms all the time and building with Tycho is the same system call than plain mvn.

If people really would do substantial stuff, then they can do it and run Maven.

I guess a greater problem is the lack of documentation in the complex internals of Xtext.

rasifix commented 3 years ago

I agree that lack of documentation and internal complexity are certainly also bad. Regarding Tycho: you shouldn't scare people away when it is not necessary (e.g. building eclipse plug-ins). You are the ones that try to get new people on board, not me ;-)

drkstr101 commented 3 years ago

@cdietrich

I actually have a fairly deep knowledge of Gradle, up to 6.7. Are there any open tickets I can take a look at? I will search eclipse-xtext for any matches, but let me know if I should I look elsewhere.

Sadly, I probably will not be able to make much progress on MultiQuickfixTest API until I can afford an upgrade to my machine... Soon I think! =)

cdietrich commented 3 years ago

@drkstr101 see https://github.com/eclipse/xtext/issues/1548

ThLeu commented 3 years ago

Hello everyone and first of all thank you for all the hard work on Xtext and also thank you on bringing this topic up and opening the discussion, sadly I only saw it today and read through all of it and wanted to give you another perpective.

As I don't know Xtext for a very long time and also do not know all of it's history I would just like to give you one more perspective from a company that uses Xtext and how we got there.

Some time ago we did an evaluation of different tools / libraries to build a (at that time) standalone DSL to provide a simple way for our clients to customize some aspects of our software. During the first stage of evaluation we already knew about Xtext but only knew it from earlier as heavily integrated with the Eclipse IDE which was an immediate no-go for us. But during the actual evaluation of Xtext we were pleasantly surprised to see that a lot of things happened and that it was actually split into different parts (most importantly xtext-core). Hadn't there been an xtext-core component at that time, we might have used something different, or written something on top of ANTLR ourselves (for the use case at the time). Today we are glad we decided on Xtext and are using more and more features to provide a feature rich DSL. But all of those features we care about are right there in the xtext-core repository.

I would agree with many of the already mentioned statements, most importantly with the following aspects:

Some additionals remarks

So long story short, these are my main points:

All of the above would already help to make it easier and more attractive for contributors.

Lastly, thank you again for all the hard work and I hope that I will myself be able to contribute something in the future, even if it might only be smaller things like documentation or tests etc. as I am missing crucial knowledge to be able to contribute more at the moment.

Best regards!

schwichti commented 3 years ago

For me, the template expressions in Xtend are by far the best concept for string interpolation. If it didn’t exist, I would never have started CrossEcore.

As some have already mentioned, I also don't think that the build system is the decisive criterion that scares off potential contributors. Personally, I see it that way that it would be best to invest knowledge building in Gradle. In my opinion, this also looks best on a resume. Maybe others think so too. Ultimately, I don't care about the build system.

I suggest to disentangle Xtext from the Eclipse ecosystem wherever possible, simply to make it accessible to a wider audience. Hosting the code on GitHub and using GitHub as the main communication channel was a good decision.

kdtp commented 3 years ago

I am new to Xtext. Though I am a seasoned developer, it is quite hard to dive into Xtext and its framework. After completing the tutorials, watching many videos there are too many open issues. This let me turn away from even considering being helpful in future Xtext development. And I even have to reconsider my decision to use Xtext in production.

So in short: For a living project you need people.

I love the ideas I have seen in Xtext, I choose Xtext over other solutions. But I have got stuck too soon. For me it seems being a seasoned developer is not enough to get involved. I have to learn a lot about other eclipse frameworks, and there is even no hint where to start.

So in short: You need tutorials, videos, whatever beyond some beginner tutorials

I am not giving up now, but the clock is ticking.

cdietrich commented 3 years ago

see also https://github.com/eclipse/xtext/issues/1961

JeffLowcock commented 3 years ago

I am an experienced developer and it is clear that without documentation it is nigh impossible to grapple with the code base. One immediate suggestion would be, to a certain extent, parallel the structure of eclipse itself. Trawling through the code is an almost fruitless endeavour for someone who does not live and breathe it. As an example I found interfaces and classes that deal with various resource across many projects so something like org.eclipse.xtext.core.resources would be nice, same for text editors, etc. This would ease some of the introduction to the code base because the alignment with current eclipse platform projects would allow easier identification of the probable location for the code and would help someone with more generic eclipse knowledge to guide themselves towards the likely project containing that concept. My personal experience love the tools, but OMG if you want to do anything outside of the generated solution you are pretty much on your own.

mundacho commented 2 years ago

I've been working with Xtext professionally for 7 years and currently work on it daily. I've some free time that I'd be willing to contribute. I've dug in Xtext code base a lot to debug issues, but a bit of help to get started and direct my efforts where they can contribute the most would help a lot...

kthoms commented 2 years ago

@mundacho Help is everywhere needed and appreciated. This can be help fixing bugs, observe and test dependency upgrades, support latest Java/Gradle/Maven/LSP/Eclipse, documentation, supporting users here and on the forum, review PRs ,.... It depends on where your expertise and interests are. Just contribute where you think it is for you or the users helpful. Of course there is always a list of issues to look at. And maybe you have your own ideas and just realize a cool new feature.

cdietrich commented 2 years ago

also helpful besides working on bugs/features is also the "paying attention" work. e.g. updates for orbit, other dependencies, tycho etc.

jmiguelhdez commented 2 years ago

is xtend still needed?

cdietrich commented 2 years ago

there is no good replacement for

RBirdwatcher commented 2 years ago

Personally I think there is a lot of good documentation on Xtext. There are other projects like PlantUML which try to go from simple grammars, to some useful tools like visualizing models. Xtext to me is the most fully functional project for having a useful model linked to a grammar..There are some competitors, and XText perhaps can be pointed to as having lots of user interaction on the forum historically and still (combined with many articles). When trying to convince others to use Xtext and not a competitor, I like to point to this usage and forum discussion compared to its competitors......but unfortunately the Forum starts with a Pinned message that the future maintenance of XText is at risk....I understand thiat this raises awareness, but it is not a good look when trying to encourage use of Xtext over other less mature projects........is there a point where we may consider this message could be taken down/unpinned? I have seen lots of good discussion in this issues over 2 years...maybe that message pinned on the forum is no longer needed?

cdietrich commented 2 years ago

@RBirdwatcher the problem stated is still true and the situation gets worse and worse. so do you have any proposals the get the problem solved?

mundacho commented 2 years ago

Hello,

I've been closely following the Xtext development since the beginning of the year and I'm aware of the need for someone to do the maintenance. I just saw a message that the problem continues, and I think there is a solution: Open Collective.

Open Collective is basically a website to help fund open source projects. People can contribute money, and then it is paid to the people who actually maintain the software. I've seen it work with a web framework I used that was in exactly the same situation as Xtext, Playframework. Basically, the company that created it (Lightbend) stopped maintaining it, and they decided to setup an Open Collective account for the project so that the community can pay for its maintenance.

Now, companies and individuals using it are donating to the open collective and payments are being made to a contributor that maintains it. Here is the website of that project http://opencollective.com/playframework . It seems have worked for them and the situation seems to be improving. At least they are now having regular releases.

Maybe Itemis or the Eclipse foundation (I don't know who has to do it) can do this for Xtext, and leave it in the hands of the community. We just need someone willing to maintain it, maybe @cdietrich if he wants and can (since he is the one who does it now). I would happily do it, I've been trying to contribute to the project for a while, but in my free time there is a limit to what can be contributed to a project as large as Xtext. Maybe there is also someone else with a lot of experience who can volunteer.

What do you think?

ewillink commented 2 years ago

Open Collective seems interesting. A step towards what's needed, provided those with money also have consciences.

The Collective must be Xtext (or OCL or UML2 or ...) to ensure that the money goes to the project. A bitter experience with occurred when a generous donor did make a significant donation to 'rescue' QVTo, the donation was swallowed by Eclipse and no QVTo committer saw any specific benefit other than the generic EF support for GIT / Jenkins / ...

I gather that the EF is experiencing almost record membership so I'm unclear why there is no EF trickle down funding, at least for significant and under threat projects such as Xtext.

cdietrich commented 2 years ago

@ewillink this is basically the same information we got from the foundation when we evaluated the possibilities years ago. They don’t want projects to be directly sponsored

ewillink commented 2 years ago

I don't follow 'same'. The EF has no problem with direct project sponsorship whenever a committer is a paid employee of a company so I see no reason why a project should not distribute its income to its members. If necessary set up a nominal company whose directors are the committers and whose bank account receives the donations and distributes to directors / expenses. I just caution against letting the EF get its hands on donations since my experience is that our money vanished.

lppedd commented 11 months ago

I've had a read of all the discussion here.
I'll be short in explaining why people don't contribute to Eclipse projects involving PDE (but even to some that don't), with just three points.

  1. p2. Plugins. Features. a.k.a complexity. I work with PDE (and RCP) since ages and it's always been a pain.
    Each time I work with plain Java project my stress level gets back to zero. Also, this technology is becoming niche, if it isn't already.

  2. Eclipse IDE is pretty much mandatory. No Eclipse, no p2, no party. Now I bet some of you will say "We've got Tycho, Maven! Yay!". Well unfortunately Tycho doesn't play well with any other IDE, as it's pretty much another version of Maven. And I know it to the extent I've built my own IntelliJ IDEA version to support it. You can build through the CLI, but importing a project and getting the IDE to know what the heck it's editing is impossible right now. New engineers want VS Code, IntelliJ, the cool stuff, and they're willing to reject job offers to avoid Eclipse, that says a lot.

  3. Setting up an Eclipse development environment is a mess, or an hit and miss at best.

The strategy for Eclipse projects should be to extract all the core functionalities to plain Java projects (Maven, Gradle, whatever is cross-IDE), and inject those dependencies in p2 offerings when needed. This allows core contributors to focus on plain Java. Eclipse lovers to keep their Eclipse integrations up to date. Same for whatever other community-driven IDE integration.

DevEx should be the primary focus, always.
And sometimes DevEx means dropping stuff that causes pain on every code change just for the sake of backward compatibility.

An additional note: IntelliJ IDEA Ultimate supports LSP now.

Michka1111 commented 6 months ago

I've not read all these 77 comments written in ~3 years, meaning ~2 per month, this is not so much. I began with Xtext a month or so ago, let me say that I found the framework Xtext (and Xtend) marvelous, and am very enthusiastic to start my DSL with. I do not agree with so many thoughts against Eclipse. The integration of Xtext within Eclipse contributes to its richness. The Eclipse framework appears to be useful and stable, and it works. Just I would like to know what is happening in the background a little more. I use the EDSL 2023-12 Eclipse Package and review the examples, the "7 languages in 7 days" along with those of the "Implementing a DSL with XText and Xtend - 2nd Ed." and I'm able to do the projects as explained. So, in my opinion, Eclipse is not the reason why contributions are decreasing. The fact is that now I've done the samples, I want to start my DSL. And at this point, the pain begins. I'm stuck. I know how to write the grammar, I know implement tests, validators, compilers, interpreters, write Xtend code, etc. However, I do not know how to use the projects .ui, .ide, and .sdk for my DSL. The 7 language examples show many snippets of code without referencing which sub-project and which file to put the code in. It seems that the goal is just to write code, and edit source DSL, but not compile, and not interpret, and not integrate into my application. I could write a grammar to drive an ECS System, i.e. typed entities, components, and systems with methods to call, (even Copilot does it), but after? I want to run the commands of my source DSL, how to do this? I'd like to write a grammar to collect data to load into the components, I'd like to call the methods I defined in my DSL for the ECS's systems, how to call them from my DSL? which classes to code, in which file, in which subproject to put them? Which are and where to put the "inject extensions"? It is said that Xbase comes with a compiler and interpreter, where are they? How do I use them? How to run all of these? What are the details behind the scene of a Helper? I did not see any explanation about all that. All these with no answer, in my opinion, let the enthusiasm decrease. After 40 years of computing, all the technologies that grow up are those that answer these kinds of questions. Xtext documentation has links to Javadoc pages which are just unreadable and bring nothing. I'm sure not being alone in this situation, the decrease in the contributions seems to prove it. I'm dressing up a mind map to solve this problem, but it is very time-consuming exploring the examples and the subprojects, understanding the underlying technologies, and then completing the map, but… Maybe there should be more pictures that say more than long speeches when needed. Hope this scream of heart will not rest in silence! I'm very serious: it's just about laughing, this is driving life!

miklossy commented 6 months ago

@Michka1111 Thanks for sharing your thoughts with us!

Xtext documentation has links to Javadoc pages which are just unreadable and bring nothing.

Could you please enumerate on which places this exactly happens, e.g in a new GitHub issue?

Michka1111 commented 6 months ago

From the Xtext doc: at random:

From the 7 languages:

Just to say that the more I go through the doc, no didacts, no useful comments on what I have to do, the more I am in the darkness. Clearly, I know software development (C, C++, Smalltalk, SQL, ksh, lex-awk-yacc, I want to learn Xtext, Java, Injection, Keys, all that, but I want do this through my DSL implementation, not spend hours off Xtext, learn, do other examples opposite of my DSL, and then go back to Xtext and do, even not sure after learned away from Xtext I'll know how to apply.

Even the links to Xtext files in the GitHub site are difficult to read because absolutely nothing is telling me how to apply to my DSL. How the subprojects are interrelated, and how to implement these interrelations, so answering to my fundamental questions.

cdietrich commented 6 months ago

the main painpoint here: Xtext is so flexible there is 10_000_000_000 ways of doing things....

Michka1111 commented 6 months ago

Sorry, but no, there is only 1 way I'll implement my DSL, and then run it, and please help me to choose this one way among the 10^10 ways…

cdietrich commented 6 months ago

again: depending on the usecase each of these ways will be the correct for you. for a general walktrough i can recommend you lorenzo bettinis book.

btw did you follow "the tutorial". all the rest of docs is "reference docs"

opcoach commented 6 months ago

@Michka1111 if you need to use your xtend generators in an Eclipse UI, it is possible to call them in E4 commands using model fragments.. I can give you some tips for that... And this is exactly the training I am starting today in Paris ! We have xtend generators on ecore or on a business model and we connect them using ui selection and E4 commands...

tripleo1 commented 6 months ago

My two cents:

Javadoc in itself is hard to read and mostly not useful, no matter what the project. This is based on the stylesheet(s) and the fact that there is so much more to know.

A lot of documentation is out of date, like I tried the sample projects from dsl-2023-06 or 09, and at least one did not run with the workbench/space/whatever it is distributed with.

IntelliJ EDU has kind of the right idea ;), and I'm not so sure that eclipse didn't influence it.

Michka1111 commented 6 months ago

Hello, I've read the head post of this thread, because many questions pop up to decide if I go further with Xtext or if I look for another framework. Why abandon Xtend/Xbase? In what "Xtend is no longer the Java 17 of today"? Where is the discussion for this where I could find more information? Because all the literature & examples on Xtext are related to Xtend.

Michka1111 commented 6 months ago

A little testimony: I've looked at Langium to see if I've found answers to my fundamental questions. Langium appears based on Xtext. Langium's grammar seems similar to Xtext's. But there is ONE different keyword, a keyword used daily, hourly, minutely, secondly, everywhere in the two grammar and documentation, respectively. In Xtext, this keyword is 'RETURNS'. In Langium, it becomes "INFERS"! To my knowledge, it made "bingo"! All went enlightened! Then, the schema of the Xtext parsing process popped up instantly under my pencil! This impacts deeply my understanding of Xtext. And comforts me to go further with Xtext. So, now I'm aware of the future maintenance of Xtext, despite I'm not (yet) a maintainer, but because I'm a user. I think a major question is "To whom is made Xtext for?" To its "citizen" domain expert users? Or to its Java-advanced users who speak to the domain experts? Or only to its maintainers? Is the future maintenance wanted to simplify the hard work of maintainers or to simplify the usability for citizen domain experts? In my experience, it'll be both.

ewillink commented 6 months ago

When it came out, Xtend could be viewed as a Java++ because Java had stalled. Now that Java is moving (too) fast with many developers, there is no realistic way that Xtend can remain ahead of Java. Also Java is stealing many of Xtend's good ideas, even string templates albeit with a very inferior prefix/suffix sequence.

Michka1111 commented 6 months ago

Ok for Xtend which had its glory hour, and now tends to be history. So, about this, what users have to do is not change the default value of the "preferXtendStub" in the .mwe2 workflow, right? Or are there other hidden things to do? How to migrate or refactor the existing tutorial samples and practices? (as for Newbees, this is just what they have to learn Xtext) Or are there some incompatibilities in newer Java editions when still using Xtend, for instance in the java that is generated from Xtend? What are the best practices of newer Java editions that apply to Xtext use cases?

What about Xbase? As Xbase leverages Xtext for Xtext users, I see a real added value. For instance, What about XBlockExpression or JvmTypeBuilder when remaining in pure Java? Is Xbase maintained relative to new features of Java? How to refactor?

For a user, this is not that simple, or do I miss something?

On this point, my opinion is that Xtext is evolving for maintainers rather than for (new) users, leading to an actual decrease in contributions.

cdietrich commented 6 months ago

On this point, my opinion is that Xtext is evolving for maintainers rather than for (new) users, leading to an actual decrease in contributions.

the capacity of the project is below the barrier "evolve for new users" 2 years+ before this post was created. specially about the docs. when they are insufficient. making it better would be something for a user to feed back their learnings

ewillink commented 6 months ago

IMHO Xbase is very important, and from my OCL-developer perspective I am very jealous. It is really hard for users to add OCL as an expression language for their grammars. Adding Xbase is fairly easy. Supporting 100% of Java is not necessary and actually undesirable. How many user grammars want the most abstruse Java functionalities and all their reserved word/punctuation corrolaries? Xbase should be a basic friendly familiar language. If someone really wants full Java they should develop an XJava extension.

Naum-Tomov commented 6 months ago

I think it may be relevant in this discussion to mention that Xtext remains relevant in academia. In terms of citations, there doesn't seem to be any significant decline, with 2023 even beating 2022 in terms of citations.

image image

Source: scholar.google.com

miklossy commented 6 months ago

Nice to hear that @Naum-Tomov ! Thanks for sharing this information with us!

ansgarradermacher commented 5 months ago

Dear all, I just want to let you know that the code generators in the Papyrus extension SW Designer are all based on xtend, i.e. we would need to find a migration path if xtend is no longer part of the Eclipse train (while the text templates in Java 21 are promising, they do not support control structures and the nesting of calls with proper indentation. I have done some experiments with lambda expressions to achieve loops within a text template, but it is too early to say if that would enable a smooth migration). We use xtext editors in Papyrus core, currently in combination with xtend. The dependency can partly be solved by re-generation, but there is some custom code in xtend (relatively easy to migrate). The tooling of GMF diagrams is affected as well, but less problematic as upcoming Papyrus versions will switch to Sirius-based diagrams.

LorenzoBettini commented 5 months ago

@ansgarradermacher Currently, Xtend is still part of the Eclipse train, and I personally have contributed and am contributing a lot to it. You can use Java 10 text blocks and Java 21 templates ONLY for very simple cases; in that respect, nothing in Java beats Xtend multi-line strings ;)

ansgarradermacher commented 5 months ago

Thanks @LorenzoBettini, good to know! (of course, not a guarantee for a longer term perspective)

LorenzoBettini commented 5 months ago

@ansgarradermacher WOW! Because there are things that are guaranteed forever! ;) Sorry for the sarcasm, and don't take it personally. But besides contributing almost daily I cannot provide any further guarantees.

maxweissboeck commented 5 months ago

Off Topic but worth to mention.

Lorenzo, I really appreciate your book, it helped me a lot to build a really cool Xtext application as a replacement for a "monstrous" Oracle Forms/Visio application. I can only recommend the book to anyone who is not yet an Xtext expert.

Best regards Max

Am 10.04.2024 um 10:11 schrieb Lorenzo Bettini @.***>:

@ansgarradermacher https://github.com/ansgarradermacher WOW! Because there are things that are guaranteed forever! ;) Sorry for the sarcasm, and don't take it personally. But besides contributing almost daily I cannot provide any further guarantees.

— Reply to this email directly, view it on GitHub https://github.com/eclipse/xtext/issues/1721#issuecomment-2046847251, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIRDSJ4C3QQYJCRXQOM6YNTY4TX25AVCNFSM4LWQOK72U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGY4DINZSGUYQ. You are receiving this because you were mentioned.