Keep comments in AST (and use them in DumpVisitor)

[GoogleCodeExporter]

Perhaps this is already possible, but I was not able to find how.

It seems that currently, javaparser dumps all comments into 
CompilationUnit with only JavadocComments receiving special treatment and 
being attached to their respective ClassOrInterfaceDeclarations, 
FieldDeclarations and MethodDeclarations.

If javaparser is used, for example, to write a pretty-printer, it would be 
useful if the comments were kept as part of the AST. A possible way to 
achieve this could be having a:

List<Comment> getTrailingComments()

method in every Node, which would return the list of comments following 
that node.

Once this is implemented, DumpVisitor should be updated (perhaps via an 
optional flag in the constructor) to print comments as well as other nodes.

Currently, a workaround is possible by looking at endLine and endColumn of 
the node you're at and then looking for a Comment whose startLine and 
startColumn precede the next node you would've printed. This is fairly 
ugly and inefficient.

The issue why I personally need it for is GWT (http://code.google.com/
webtoolkit/) native methods whose syntax is:

private static native void foo()/*-{
    // javascript code goes here
}-*/;

In the meanwhile I've solved it with a new token type and adding 
pushNativeMethodBodyBlock() (similarly to how pushJavadoc()) at the 
MethodDeclaration production rule.

Original issue reported on code.google.com by msa...@gmail.com on 21 Jul 2008 at 5:15

[GoogleCodeExporter]

Thanks msasha. The DumpVisitor needs to be rewrite because it don't respects the
token positions, when it has been done will be more easy to treat comments into 
the
dumper. 
the getTrailinComments method is a nice idea and can be useful in situations 
like
yours, but for now I think you have followed the best path creating a new kind 
of
token, specially because gwt uses a special syntax. 

Original comment by jges...@gmail.com on 22 Jul 2008 at 3:56

• Changed state: Accepted
• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

This feature request is indeed very important to create code formaters / pretty
printers. I'd like to have it as well

Xavier.

Original comment by xal...@gmail.com on 4 Jan 2009 at 5:53

[GoogleCodeExporter]

>>In the meanwhile I've solved it with a new token type and adding 
>>pushNativeMethodBodyBlock() (similarly to how pushJavadoc()) at the 
>>MethodDeclaration production rule.

Can You share Your code? I really need to keep comments in my code. 

Original comment by mariusz....@gmail.com on 6 Feb 2009 at 3:40

[GoogleCodeExporter]

Mariusz, this solutions seems to be a very good solution to your specific 
problem,
but I think a more generic solution will be needed to keep comments in every 
where. I
am planing to solve this issue soon as possible. I need to get more free time 
to do
this. 

But you can share your solution as a patch, so, other people who needs it will 
can take.

Original comment by jges...@gmail.com on 6 Feb 2009 at 4:06

[GoogleCodeExporter]

Sorry, I just took a quick look at it and extracting my changes is a bit too 
time consuming for me. Hopefully 
jguesser will be able to implement a generic solution soon.

Original comment by msa...@gmail.com on 6 Feb 2009 at 4:13

[GoogleCodeExporter]

Julio,

Even if you have not enough time to make this enhancement, would you mind 
sharing
your design ideas on how comments could be made at all AST node levels? Do you 
think
a change to java_1_5.jj is required?  You mentioned something about 
DumpVisitor, so
do you see possible to "just change" this visitor so as to get comments from
SpecialTokens?

Last question, but not least one (sorry): any idea when you'd have time to 
implement
a generic solution for comments? can we help?

Xavier

Original comment by xal...@gmail.com on 18 Apr 2009 at 10:09

[GoogleCodeExporter]

Xavier, 

I'm pretty sure that change the java_1_5.jj will be required. Actually, the 
comments
are already stored inside de AST, but only in the CompilationUnit node. It 
could be
enough to DumpVisitor rewrite the source code as read from the input file. But 
the
parser ignore some nodes from the input file, like visibility modifiers 
("public",
"private",...), they are exchanged by a numerical constants and their position 
inside
the source are lost. If there is any comment between two of these tokens, their
position are unrecoverable.
I think the design of the solution for this issue could be something like msasha
suggested in the opening of this issue. But the Issue 12 must be fixed first.

Original comment by jges...@gmail.com on 21 Apr 2009 at 12:52

[GoogleCodeExporter]

Hi

Having the comments in the AST tree is a feature I'd also really would like to 
have
for my project called "Unimozer" (http://unimozer.fisch.lu).

I also could give a helping hand by coding or testing some code ....

thanks a lot

Original comment by goo...@tux.lu on 10 Jun 2009 at 7:45

[GoogleCodeExporter]

I'll cast my vote here as well.  I was developing an internal java code 
transformation tool tool quite nicely with 
JavaParser (simple and elegant - I like it - good job) until I found I can't 
keep the in-code comments.  Yikes!  
Granted that this issue is now over a year I old I'm not hopeful it will be 
fixed, but ... I'll cast my vote here for it!  
I'm now reaching for my antlr manual ....

Original comment by gmccre...@hotmail.com on 7 Dec 2009 at 10:02

[GoogleCodeExporter]

Yeah, I mean my question is what would be required for this to be fixed. If 
it's not
too complicated we can take a look.

Original comment by yousuf.r...@gmail.com on 7 Dec 2009 at 11:19

[GoogleCodeExporter]

I've not yet scratched around the source to this project - so, honestly, I 
couldn't
comment.  But a "DumpVisitor" rewrite (as described above by the author) sounds
serious.  :-)  

It look like his issue is that the line/col info might be lost or muddled, but
personally, I'm not concerned with that.  I was intending to use JavaParser to
transform Java code into another language - with comments intact, and then 
'pretty
print' that non-java code - so line/col info is not necessary.

Original comment by gmccre...@hotmail.com on 8 Dec 2009 at 3:05

[GoogleCodeExporter]

Ha! So you were writing a compiler :D

Original comment by yousuf.r...@gmail.com on 9 Dec 2009 at 2:10

[GoogleCodeExporter]

By modifying the DumpVisitor class, I was able to keep comments that are inside
methods. While waiting for a nice solution, here's my quick hack-around ...

Original comment by goo...@tux.lu on 13 Dec 2009 at 8:30

Attachments:

• DumpVisitor.java

[GoogleCodeExporter]

Any news about this bug?

I think when you do any kind of automatic refactoring you need to have comments 
were 
they were.

Original comment by panbyd...@gmail.com on 18 May 2010 at 10:40

[GoogleCodeExporter]

Just another little comment to point out that there are still people out there 
waiting for a nice solution ;)

Original comment by goo...@tux.lu on 16 Nov 2010 at 5:45

[GoogleCodeExporter]

Is there any update on this issue?
This missing feature is making this parser unusable for me.  )-:
Does anybody know any other java parser available?

Original comment by hudara....@gmail.com on 8 Dec 2010 at 8:05

[GoogleCodeExporter]

Its a pity, that this isn't fixed yet. unfortunately i have look arround for 
some other parser :/

Original comment by sebastia...@googlemail.com on 16 Dec 2010 at 9:11

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Here is a solution to the issue.  Use the attached files to replace 
corresponding files.

Original comment by shelbyy...@gmail.com on 21 Apr 2011 at 6:03

Attachments:

• ASTParser.java
• ASTParserConstants.java
• ASTParserTokenManager.java
• DumpVisitor.java
• JavaCharStream.java
• Node.java
• ParseException.java
• Token.java
• TokenMgrError.java
• javaparser-1.0.8.jar
• java_1_5.jj

[GoogleCodeExporter]

Very nice shelby works like a charm. 

Original comment by achinto....@gmail.com on 10 Jun 2011 at 10:54

[GoogleCodeExporter]

I have tried the modified classes but i think there is a problem in the 
following situation:

// comment 1
if (expression)
{
// comment 2
code
// comment 3
}
// comment 4
else if
{
// comment 5
code
}

The comments 1, 2 and 5 are reported, but not 3 and 4.

Original comment by massimo....@gmail.com on 15 Jun 2011 at 10:48

[GoogleCodeExporter]

That is the limitation of the parser design.  Every comment has to be 
associated with something that is a japa.parser.ast.Node type.  For 3, there is 
no japa.parser.ast.Node type following 3, thus it is ignored.  You have to at 
least add a ";" following 3 for 3 to show up.  For 4, the entire "if - else if" 
block is considered as one statement, 4 is thus ignored and 1 is preserved.

On an unrelated issue, here is how to create multi-line comments (this is not 
possible without the changes I made): 
japa.parser.ast.LineComment comment = new 
japa.parser.ast.LineComment("comment");
comment.setComment(new japa.parser.ast.LineComment("comment of comment"); 
japa.parser.ast.expr.NameExpr expression = new japa.parser.ast.expr.NameExpr("a 
= b");
expression.setComment(comment);

Original comment by shelbyy...@gmail.com on 15 Jun 2011 at 11:30

[GoogleCodeExporter]

Here are the corresponding changes for using Shelby's solution with the 
VoidVisitorAdapter. Alternatively, users can extend & override every public 
visit() method of the VoidVisitorAdapter class to also accept() the node's 
comment.

Original comment by eliang...@gmail.com on 16 Jul 2011 at 8:45

Attachments:

• VoidVisitorAdapter_Comment-Support.diff
• VoidVisitorAdapter.java

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi, can anyone tell me how to use this VodidVisitorAdapter. I've copied to my 
project and I get a lot or errors because n.getComment() where in 
{AnnotationDeclaration, AnnotationMemberDeclaration, ...} doesn't exist.

Original comment by ellabail...@gmail.com on 15 Nov 2011 at 4:46

[GoogleCodeExporter]

Hi

You've forgot to mention your intentions. What do you want to do?

Original comment by goo...@tux.lu on 15 Nov 2011 at 4:51

[GoogleCodeExporter]

Any chance to get the solution into a release?

Original comment by timo.wes...@gmail.com on 4 Jan 2012 at 10:10

[GoogleCodeExporter]

The Shelby's solution dosen't work when there are no comment in source code.
We have to add a test in the init method of the ASTParser when comments is null.

Original comment by walid.kr...@gmail.com on 22 Feb 2012 at 2:44

[GoogleCodeExporter]

Regarding timo's question, I am not in the developer's circle of javaparser, 
thus I cannot merge my changes into the source. 

I am not sure I understand walid's question. The point for the changes is to 
preserve comments in the source code.  If there is no comment in the source 
code, then the comment of a node will be null.  Why do you need to test it in 
the init method? 

Original comment by shelbyy...@gmail.com on 22 Feb 2012 at 8:06

[GoogleCodeExporter]

I think the problem must be that other bits of code are expecting a non-null 
node. Anyone know if there are still engaged committers on this project? I seem 
to have seen that only users are currently commenting for the last two years. 

Original comment by chanoch....@gmail.com on 24 Feb 2012 at 8:59

[GoogleCodeExporter]

I'm still following this thread with big interest. I do not have any commit 
rights, but I'm willing to help as good as I can.

Original comment by robert.f...@gmail.com on 24 Feb 2012 at 9:07

[GoogleCodeExporter]

Since the creator of this project doesn't seem interested in it anymore, I've 
forked it to https://github.com/matozoid/javaparser - the patches are applied, 
unit tests fixed, code refactored a bit. I've since found Eclipse's JDT core, 
which seems to contain a bigger, but more complete Java parser, but I'll apply 
any patches you send to me :-) (Or you can fork my copy, it's fine.)

Original comment by tsuih...@gmail.com on 24 Feb 2012 at 12:10

[GoogleCodeExporter]

If you have added my changes to your forked copy, then that's all I have.  
Thanks.

Original comment by shelbyy...@gmail.com on 25 Feb 2012 at 1:06

[GoogleCodeExporter]

No problem. I vaguely remember not using null but empty list, so the unit tests 
would run again.

Original comment by tsuih...@gmail.com on 25 Feb 2012 at 1:18

[GoogleCodeExporter]

Hello everybody.

Sorry, I don't have time keep updating this project. I really enjoyed working 
on this thing, but is hard for me act as an active commiter.

Feel free to fork this project. Sometimes I'm going to answer some questions or 
write some code, but in a very slow manner. 

Original comment by jges...@gmail.com on 25 Feb 2012 at 1:23