differences between Spoon and JavaParser

[monperrus]

Spoon and JavaParser can both be used for analyzing and transforming Java source code.

Comparison about common features:

Aspect	Spoon	JavaParser
Declaration	Native navigation from usage to declaration	Supported in the symbol resolver
License	Double license MIT or CeCILL-C	LGPL License or Apache License
Comments	Stored and attached to elements	Stored and attached to elements

Unique features in Spoon:

• Spoon provides a way to semantically analyze code elements (methods isSubType, isOverriding)
• Spoon provides a way to manipulate library classes as normal Spoon object (so called shadow classes built using runtime reflection)
• Spoon provides a statically typed template engine
• Spoon allows to compile the transformed source code on the fly.
• Spoon gives you paths to uniquely identify source code elements

[msteinbeck]

• JavaParser is a way more faster in generating an AST than Spoon.
• Again, Spoon supports navigation between usage and declaration, a feature very important to us :)
• Spoon provides a quite sophisticated query engine. Does JavaParser provides this kind of feature?
• Spoon's AST is separated into an API (a set of interfaces) and an implementation. The API is self-contained making it easy to extend the AST if needed. As far as I can see, JavaParser provides an implementation for its AST only.

[msteinbeck]

• Spoon allows to compile the source code. Very nice in conjunction with AST-transformation.
• JavaParser has a way more stars than Spoon :P.

[monperrus]

JavaParser has a way more stars than Spoon :P

That's indeed a problem :-) Let's work hard to become #1 :-)

[msteinbeck]

• ~JavaParser has a better comment parser. I would like to see a parser for JavaDoc in Spoon too. That is, I would like to have AST-nodes describing, for instance, the @param tags of comments.~ (see #1310)

[gnom7]

JavaParser has a way more stars than Spoon :P

That's indeed a problem :-) Let's work hard to become #1 :-)

It might be a good idea to add topics/tags for this project just like it was done in JavaParser (code-generation, code-generator, syntax-tree, code-analysis, abstract-syntax-tree) to make it easier for people to discover spoon.

[monperrus]

done. thanks for the suggestion.

[msteinbeck]

How about adding Spoon to the "External Links" section in wikpedia articles, e. g., https://en.wikipedia.org/wiki/Abstract_syntax_tree

[monperrus]

please go ahead!

--Martin

On 10/12/17 9:19 AM, Marcel Steinbeck wrote:

How about adding Spoon to the "External Links" section in wikpedia articles, e. g., https://en.wikipedia.org/wiki/Abstract_syntax_tree

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/INRIA/spoon/issues/1303#issuecomment-336042682, or mute the thread https://github.com/notifications/unsubscribe-auth/AAxDUsRIbnqedH_gs8LwEJB4kvOn9kYdks5srb16gaJpZM4Ndijk.

[msteinbeck]

I added Spoon to the following articles:

• https://en.wikipedia.org/wiki/Abstract_syntax_tree
• https://en.wikipedia.org/wiki/Program_analysis
• https://en.wikipedia.org/wiki/Program_transformation
• https://en.wikipedia.org/wiki/Parsing
• https://en.wikipedia.org/wiki/Static_program_analysis

[monperrus]

Well done! Thanks

--Martin

On 10/12/17 10:10 AM, Marcel Steinbeck wrote:

I added Spoon to the following articles:

• https://en.wikipedia.org/wiki/Abstract_syntax_tree
• https://en.wikipedia.org/wiki/Program_analysis
• https://en.wikipedia.org/wiki/Program_transformation
• https://en.wikipedia.org/wiki/Parsing
• https://en.wikipedia.org/wiki/Static_program_analysis

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/INRIA/spoon/issues/1303#issuecomment-336053986, or mute the thread https://github.com/notifications/unsubscribe-auth/AAxDUmgZ-o1_VGWtXdu4KthpQGZ-0icfks5srcl3gaJpZM4Ndijk.

[monperrus]

even closed, don't hesitate to comment!

[monperrus]

Spoon now supports Java 9

[vidyat11]

Using JavaParser Some comments are not always stored (missing/overwrite previous) ..how about spoon whether the comments will retain or have same issues

[monperrus]

Yes, see command "[-c|--enable-comments]"

[vidyat11]

thanks for your reply

[pkriens]

I needed a parser for Java to do some transformations. I started with Spoon but then started testing. It turned out that JavaParser did not have any transitive dependencies and I like that a LOT. So it seems Spoon has more functions but, for now, the transitive deps weigh more heavily for me.

And of course, Javaparser is OSGi ready and Spoon not :-(

[monperrus]

Another answer "Eventually we decided to settle on the Spoon library:": https://medium.com/@Coder_HarryLee/development-of-a-new-static-analyzer-pvs-studio-java-f92f7c139362

[Symbolk]

• JavaParser is a way more faster in generating an AST than Spoon.
• Again, Spoon supports navigation between usage and declaration, a feature very important to us :)
• Spoon provides a quite sophisticated query engine. Does JavaParser provides this kind of feature?
• Spoon's AST is separated into an API (a set of interfaces) and an implementation. The API is self-contained making it easy to extend the AST if needed. As far as I can see, JavaParser provides an implementation for its AST only.

I'd like to explore more about "navigation between usage and declaration", could you please provide some helpful materials?

[pvojtechovsky]

Thank you for that question ;-)

There are 2 types of nodes in Spoon model:

• CtXxxx - for example CtType. See http://spoon.gforge.inria.fr/structural_elements.html
• CtXxxxReference - for example CtTypeReference. See http://spoon.gforge.inria.fr/references.html

Each CtXxxReference has a method getDeclaration(), which returns referred CtXxxx node.

In other words. Whenever there is a usage of (reference to) type, parameter, variable, ...

someField = 7;

then it is easy to navigate to it's declaration.

int someField;

[Symbolk]

Thank you for that question ;-)

There are 2 types of nodes in Spoon model:

• CtXxxx - for example CtType. See http://spoon.gforge.inria.fr/structural_elements.html
• CtXxxxReference - for example CtTypeReference. See http://spoon.gforge.inria.fr/references.html

Each CtXxxReference has a method getDeclaration(), which returns referred CtXxxx node.

In other words. Whenever there is a usage of (reference to) type, parameter, variable, ...

someField = 7;

then it is easy to navigate to it's declaration.

int someField;

Thanks for the reply, that is a really useful feature for me. BTW, can anyone show some code examples as tutorial for this feature?

[jinfuchen]

Spoon is a nice tool to parse Java code. BTW, can I use Spoon to do program slicing?

[monperrus]

In theory yes. Do you mean static or dynamic slicing?

[jinfuchen]

Static slicing. Because I used WALA to perform program slicing before. However, the constant propagation would ignore the constant assignment statement so that I can not handle this situation.

[monperrus]

Yes, it is possible to use Spoon for static slicing. There is no dedicated support for this, but you have the ability to easily navigate between variable usage, assignments and variable declaration.

[jinfuchen]

Thanks for your reply. And I will try to use Spoon to perform static slicing :).

[ftomassetti]

Well, I am a JavaParser contributor and it seems to me that some statements about JavaParser are wrong.

JavaParser has been integrated with JavaSymbolSolver so it can now permit to travel from a usage of a symbol to its definition. It can also permit to do some analysis, for example to see if two types are compatible or to find the type of an expression.

Regarding comments, we store all comments and then we associate some of them to the thing being commented by using some heuristics.

JP was born to be just a parser and to be extremely easy to use. When you need just a parser we believe that JP is a very good choice because it is extremely easy to use, it is fast, it has been tested on millions of files, it has a very large user base and we support the newest versions of Java as fast as they keep throwing them to us.

Symbol solving is something we added over time and it is not as mature as we would like it to be. People have used it and are using it to transforming code (especially since we introduced lexical preservation) and do all sort of analysis. It has bugs, it has an API that is not obvious because of the nature of the project (it is first of all a parser and symbol solving is something optional). But all in all it is being used quite successfully.

There are also quite big projects out there that depends on JavaParser. Just a couple comes to mind: the newest version of Drools is based on a fork of JavaParser and the repository of Gradle contains some usage of JavaParser.

Spoon is something that I wanted to study for a while as I kept hearing it over and over. In general I hear about a lot of great stuff being done at INRIA. Of course I could be biased by having some French friends :) I would love to write an article on Spoon for my blog on language engineering and I would be very happy to receive some help on this, if you are interested into that (maybe it could have spreading the voice: I have some decent readership, this month we went close to the 100K views and I have a newsletter with almost 6K subscribers).

Of course I am biased but I think you are selling JP a bit short in your comparison. I am the first one to admit that some features are not mature or complete and there are some bugs but I think that your analysis could be misleading for some readers.

It is true, we have more stars :) But more stars are not just stars, more stars mean more feedback, more opinions, more contributions, more tests, more questions. People have used JP to do all sort of crazy things and this meant we got very valuable feedback and error reports.

So, I hope you do not mind for me commenting on your project. I think we are building open-source tools, doing that by sacrificing our own free time and it is hard to keep working on them. Who knows, perhaps one day we could find some way of collaborating by sharing ideas, use cases or even code. In the meantime have fun building great software!

[ftomassetti]

...oh, and another difference is of course the license: as far as I understood Spoon is released under the CeCILL-C FREE SOFTWARE LICENSE AGREEMENT, which could be maybe well known in France but it is a bit of an headache of users in the rest of the world because they have frequently never heard about it, so they need to figure it out, understand its terms, understand if it is compatible with other components they are using, etc. This has a cost in any industrial project. JavaParser is instead available under well-known and understood licenses. It may seem a minor thing but for some users it is a dealbreaker

[monperrus]

Welcome @ftomassetti ! It's great to have you here! I'm sure we'll have a super interesting conversation.

But before answering to specific points, don't take me wrong: JavaParser is great. It's mature and used by many people to do awesome things!

[monperrus]

JavaParser has been integrated with JavaSymbolSolver [...] symbol solving is something we added over time

Added in the comparison (at the top of this thread: https://github.com/INRIA/spoon/issues/1303#issue-229276369)

[monperrus]

Regarding comments, we store all comments and then we associate some of them to the thing being commented by using some heuristics.

That's a great feature that is the main inspiration for the corresponding one in Spoon, added in the comparison.

[monperrus]

JavaParser is instead available under well-known and understood licenses. It may seem a minor thing but for some users it is a dealbreaker

Fully agree! Added in the comparison.

[monperrus]

some statements about JavaParser are wrong.

We apologize for that. As you see, this comparison is meant to be factual and accurate, and I keep updating it. What's still wrong?

[monperrus]

I would love to write an article on Spoon for my blog on language engineering and I would be very happy to receive some help on this

Thanks a lot Federico for the proposal, that's really much appreciated. We'd be very happy to co-author it or to simply give you info & feedback if you prefer this way!

[pvojtechovsky]

I am also glad to see you @ftomassetti here!

I think we are building open-source tools, doing that by sacrificing our own free time and it is hard to keep working on them.

yes! I am sure we can cooperate. And it can bring a lot to both communities. I personally (I am not from INRIA ;-) ... but I love what INRIA did!) need a tool which solves my needs. And I do not care what is it's name or history. If there would be a way how to merge Spoon and JP together I would love it. But actually it seems to me like there are some deep architectonic differences which forces us to spread our effort ... and to implement thinks twice ... that is not good :-(

So there is no reason for any fight ;-). I welcome you!

[ftomassetti]

Spoon provides a way to navigate from usage to declaration (method call node to method declaration node, see getDeclaration())

You can do this in JavaParser by calling resolve on different AST nodes. For example, to find the method declaration associated to a method call you just call MethodCallExpr.resolve

[ftomassetti]

Spoon provides a way to semantically analyze code elements (isSubType, isOverriding)

You can do some of this in JP. For example, you can use Type.isAssignableFrom. We should add the isOverriding feature.

[ftomassetti]

Spoon provides a way to manipulate library classes as normal Spoon object (so called shadow classes built using runtime reflection)

I am not sure what this means, but JP can work with classes obtained through reflection, through JARs or weird Android packages

[ftomassetti]

Spoon provides a statically typed template engine Spoon allows to compile the transformed source code on the fly.

True, JP has no integration with a template or the compiler. We joked about building a compiler at some point as basically the hard work (symbol solving) is done. But probably that would not be a smart idea for us :D

[ftomassetti]

Spoon gives you paths to uniquely identify source code elements

Not sure how to interpret this, but if you solve a reference to a piece of Java code of course we can tell you where the file is, and the position inside that file

[ftomassetti]

I cannot comment on the list of things that only JP can do because I am not sure I know Spoon well enough. However that ??? made me a little sad :)

[msteinbeck]

You can do this in JavaParser by calling resolve on different AST nodes. For example, to find the method declaration associated to a method call you just call MethodCallExpr.resolve

Spoon, in order to generate semantic edges between nodes, compiles the source code using the eclipse compiler. As far as I know, JP does not compile the source code (therefore being significantly faster than Spoon). Is JP capable of resolving edges as accurate as Spoon is?

[matozoid]

I'll just add why I think JavaParser has all those stars (I'm the maintainer):

• the license. Many companies are very picky about these.
• the name. It's a very old library and it got to pick the most obvious name :-)
• the activity. Weekly releases with always something in them gives the impression of a very active project.

[ftomassetti]

You can do this in JavaParser by calling resolve on different AST nodes. For example, to find the method declaration associated to a method call you just call MethodCallExpr.resolve

Spoon, in order to generate semantic edges between nodes, compiles the source code using the eclipse compiler. As far as I know, JP does not compile the source code (therefore being significantly faster than Spoon). Is JP capable of resolving edges as accurate as Spoon is?

I am not sure what edges are :) Probably not. After all it will be hard to build something as good as the eclipse compiler. Basically we do symbol solving on demand. It means we can:

• be a bit error tollerant
• work only on subpieces of the AST
• customize the process as we want (we would love to support Java extensibility at one point)
• potentially work with multiple versions of Java
• deal better with incomplete/missing pieces (e.g., missing libraries or source code) and even mock stuff easily by playing with our typesolvers

So yes, it means it is not as good as compiler developed by a large team of talentend developers, but is way faster, simpler to use and we can control it.

[msteinbeck]

I am not sure what edges are :)

With edges I mean declaration relations between AST nodes. To resolve declarations, you either need a compiler, or you have to make cuts while resolving particular declarations. However, from what you say, JP's symbol resolver is very powerful and you almost already implemented your own compiler :).

[pvojtechovsky]

@ftomassetti, @matozoid @monperrus , ... It seems to me like it is time to move from this issue #1303 to some place where we can give our clients (I mean Spoon and JP clients) mature and fair overview about features of our two projects.

And I am personally interested in JP features too.

It is hard to give them such overview here in Spoon issue - there are many limits here ... I tried to adapt root description of this issue using your comments, but I failed ...

Do you have some idea, which tool / place to use so both sides may

• collect list of features
• discuss each feature so we have same understanding on both sides
• publish result of that work somewhere - the best is some independent place or we can have it on web side of both projects.

How to reach fairness? The sentences like "Spoon provides a way to semantically analyze code elements" and "JP can do it too" ... we might discuss forever and client's will get no overview.... ;-)

Any ideas?

As a client I would like a table with real test cases ...

1) an description of "common feature" 2) an example of to be analyzed code 3) an code snipped, which shows how to use such feature in JP in one column and in Spoon in second column.

If it is good idea, then our Spoon and JP clients might be itself motivated to contribute examples and code snippets from both sides. It will help them to see, which tool is good for which case. And it might be special kind of "self made" documentation.

WDYT?

[ftomassetti]

hi @pvojtechovsky I am not sure it would be possible to present a fair comparison between the two products. First of all we are all strongly biased as we have poured countless hours on our own projects. Then we have different opinions and point of views: what matters in an academical context is different from what matters in an industrial context. What "simple to use" means for a person is different to what it means for another person.

I think to discuss and compare features could be helpful to us contributors to see how we could improve our work, but it would be very difficult to provide a reasonable comparison that we agreed upon and it is useful for users.

I also think that we could collaborate on spreading the voice about the existence of these tools: the main problem is not that people are using SPOON instead of JavaParser or viceversa, but that people are unaware of these tools and what they enable to do.

Another thing we could share ideas is describing use cases for both tools. It is hard for us to find out what people are using JP for. We know we have thousands of readers of our book, we see it used in some repository and from time to time we talked with developers from some companies using it but the 99.99% of users just get it, use it and never talketo us.

Also, I am curious about your project: is it mainly developed by researchers at INRIA? Is it used in industrial projects?

Sorry for the confused mess of unrelated topics in one message :)

[ftomassetti]

I would love to write an article on Spoon for my blog on language engineering and I would be very happy to receive some help on this

Thanks a lot Federico for the proposal, that's really much appreciated. We'd be very happy to co-author it or to simply give you info & feedback if you prefer this way!

Ok, let me prepare a set of questions about Spoon. Then I will share them with you and ask your help to work on a tutorial. Who want to participate? Should we move the discussion on the article to emails?

[pvojtechovsky]

Hi Federico

I am not sure it would be possible to present a fair comparison between the two products. First of all we are all strongly biased as we have poured countless hours on our own projects. Then we have different opinions and point of views: what matters in an academical context is different from what matters in an industrial context. What "simple to use" means for a person is different to what it means for another person.

I afraid you are right.

I think to discuss and compare features could be helpful to us contributors to see how we could improve our work,

I fully agree. I welcome any suggestion about how to do it.

I also think that we could collaborate on spreading the voice about the existence of these tools: the main problem is not that people are using SPOON instead of JavaParser or viceversa, but that people are unaware of these tools and what they enable to do.

Also, I am curious about your project: is it mainly developed by researchers at INRIA?

I guess @monperrus may answer that the best.

Is it used in industrial projects?

Yes, it is. I am not from academic area but from a company which develops software. I already have some very nice code generators based on Spoon and I would like to do some deep refactoring/cleaning of our source code. Here I need full power of Spoon (100% correct AST model with full java semantic. powerful query engine, detection of patterns, pattern matching, refactoring, and printing of changes only including correct imports, etc.). Before I found Spoon, it was kind of utopic idea. I would say it is impossible. But now I see it is doable with Spoon.

I gave JP only short look and it seems to me like it is very hard to build correct AST model with full java semantic without a compiler. So it is not sufficient for my needs. But it is visible on number of JP starts, that most of the clients does not need a "powerful rocket" (Spoon). It is much easier and often sufficient to have "just" ;-) a "Lamborghini" (JavaParser).

Sorry for the confused mess of unrelated topics in one message :) Should we move the discussion on the article to emails?

@monperrus what do you suggest? I guess you want to attend too ;-) I would prefer to keep this discussion open ... but may be on different Github issue?

[Evestar7]

Latte with an espresso shooter. Que Rico Suave. Como que Suave.

[tdurieux]

Also, I am curious about your project: is it mainly developed by researchers at INRIA?

For the history, the spoon project started at the beginning of 2007 with Renaud Pawlak as a tool for research (if I'm not wrong @monperrus should better now that). The spoon architecture did not massively change since that period. But starting 2008 until 2012 the project was mainly inactive, and in 2014 a full-time engineer was paid for two years by Inria to make to project more stable. This contract was followed by a second 2 years contract to maintain spoon. Currently, there is no paid engineer on the project and it is maintained mostly in free time. I would say that the active community of Spoon is half academic and half industry.

Since 2014, @monperrus used Spoon for massive code instrumentation (meta-programming, automatic repair, fault localization, ...) in full classpath. Then 2/3 years ago, we started to work on supporting a noclasspath mode, in order to be able to analyze and transform code without having the full classpath. With @pvojtechovsky, we have now a powerful query engine and template matching system and did a massive job on spoon to allow automatic local change (sniper mode #1927). That will open new perspectives. (I really don't know the position of JavaParser about those features).

Another thing we could share ideas is describing use cases for both tools.

I think we also have this issue to know which is a typical usecase for the project. But I think, our tools are so generic that we can almost do anything.

For me the use cases are:

• code analysis (metrics, behaviour, ...)
• code check (we use spoon on spoon, for quality control)
• code generation (we generate some visitor with spoon, or some statement and annotation has been automatically added)
  • fun (for example I did a transpiler C to Java, using the Spoon AST)

But you probably already faced those use cases