comment-line related problems.

[GoogleCodeExporter]

Hi,
I have two things.
One is my own change (A), the other is bug reports (B).

As for (A), I now want to keep fortran code as it is in self.item.
In order to do so, I made a little modification to readfortran.py.
If possible, I hope you will include this as an option in your fparser.

---preparation to reproduce my results---
In a directy, expand IssueCommet.tar.gz.
Then copy fatest.py just above f2py.
Then replace f2py/fparser/readfortran.py (r25) with readfortran.py in
IssueCommet.tar.gz. (Try diff of these two readfortran.py.).

==== (A) Changes in readfortran.py======
Perform
  fatest.py CommentTest.F.
There is some difference when we use original readfortran.py (r25).

1. lines with '#' at 1st column are recognized as comment lines.
   As far as fortran codes are grammatically correct even 
   when "#foo"s are treated as comments, we can analyze the codes
  (I rewrote my codes so).

   Related to this, I show my opinions below.--

    In future, I hope that fparser can recognize #foo as 
    fpp directives (especially it is necessary to distinguish #if block
    structure). In order to do so, it might be better that 
    not FortranFileReader but FortranParser should identify CommentBlock
    from others on the same footing as other block_statements.
    Inline comments (starting from !) can be an attribute of each line.

    I think CommentBlock is not so good identification.
    Fortran comments can be an attribute of a line --- 
    then we can treat inline comment and traditional comment lines 
    on the same footings. Class of a traditional comment line may be 
    named as DoNothing.
    (Then we need another attribute --- span of the comments). 
    It is not so meaningful to distinguish two comment lines;
    one starts from "!" at the first column, and the other starts 
    from second column.

2. I added "\n" at the end of CommentBlock.comment.
   So the output of fatest.py looks like item=Commnt('...\n',span).

3. Even a line with nothing is recognized as a comment line.
   So we see 
     item=Comment('c test abb\nc\n# 1232\n!\n\nc\ncabi\n',(3, 9))
     item=Comment('\n',(14, 14)) 
   for example.

===(B) Bug reports===
1. Perform
  fatest.py Issue_r25_1.F 
 Then I can see item=Comment('!hhhh2\n!hhhh2\n',(12, 12)).
 This looks strange.

2. Perform
  fatest.py Issue_r25_2.F 
 This caused a stop (this occurs when a nothing line is in the middle of
format).

best regards,
takao kotani

p.s. 
I would like to set up a Fortran-code analyzer 
(Module dependency, call-caller tree, recursive input/output check of
subroutine. 
 Block analysis. Data flow, or so.)
The tree =
parse(ffile,isfree=False,isstrict=False,ignore_comments=False,analyze=True)
can treated as a database. It can be pickled, 
and refreshed when a fortran file is modified.

I should apply some operation to do analysis based on the database.
or I hope that you will start such a project 
you as a leader(but maybe too busy to you?).

Original issue reported on code.google.com by TakaoKot...@gmail.com on 21 Mar 2010 at 10:17

Attachments:

• IssueComment.tar.gz

[GoogleCodeExporter]

Hi,
r25 failed to parse this also.

      subroutine gwinput_v2x(ifi,konf,ncore)
      integer :: ifi, !File handle. Write
     &  konf(0:lmxamx,nclass),! Principle
                              ! For examp
                              ! Core orbi
                              !   1, 2,..
                              !   2, 3,..
                              !   3, 4,..
                              !
     &  ncore(nclass)   ! ncore = \sum_l 
                        ! Number of diffe
      end

Original comment by TakaoKot...@gmail.com on 21 Mar 2010 at 11:36

[GoogleCodeExporter]

FYI, the parsing failure in now fixed in svn.

Original comment by pearu.peterson on 21 Mar 2010 at 5:50

[GoogleCodeExporter]

I have fixed other bugs too in svn.

CommentBlock is now gone. Instead, Comment is used to
represent a comment line as well as blank line.

Original comment by pearu.peterson on 21 Mar 2010 at 10:35

[GoogleCodeExporter]

Thank you so much.

I like this. I like a simplest and a monotonic way as possible. 
I have three points below.

1. The class Comment should have self.startcolumn (or so),
   which is the starting column of the Comment.

2. item=Line(...) should contain information how a line is divided into lines.
   e.g. Line.line can be a list. Then I guess that it is not so difficult 
   to pass it to a parser. Then we can easily reproduce original codes as it is
   (except a character for continuation line).

3. This subroutine caused an error. When ! is at the second column.
------------------------
        subroutine m_struc_def()
 !abc test
        end
------------------------

regards,
takao

Original comment by TakaoKot...@gmail.com on 22 Mar 2010 at 11:06

[GoogleCodeExporter]

The bug in comment 4, point 3 is fixed in svn.

Regarding issue 8, point 1:
I am hesitating to interprete # as a comment line. Currently
fparser issues a warning because the Fortran 2008 (an earlier) standards
say that only C, *, ! in the first position of a line mark the start of 
a comment line. We could extend Fortran standard for fparser but I would
avoid that because following the standard has a value.
What could be done is to introduce an option to suppress such warning.
So, I would leave # for preprocessor to handle.

Regarding issue 8, point 2 and 3:
Since CommentBlock is gone then no comments.

Regarding issue 8, last comments:
Yes, I am too busy:) However, supporting pickle in parse tree is
a good idea. I'll look into it later. Btw, is the speed of parsing an issue?
How long it typically takes to parse your Fortran codes? 

Regarding issue 4, point 1 and 2:
What is the point of reproducing literally the original code?
It is quite difficult problem and I am not sure if it will be so
useful that the effort would pay off.
I think fparser must reproduce exactly the original code in terms of
interpretation. Comments are for completeness.

Finally, in future, could you create for each point a separate issue
so that it would be easier to track these points. Don't worry about
creating too many issues, they can be always managed.

Original comment by pearu.peterson on 22 Mar 2010 at 10:51

[GoogleCodeExporter]

At first, when I have a request (not bugs) to your code, where is a 
suitable place to post. Group or Issue? 

I think fparser is very different from other fortran checkers.
It is a kinds of database generator, words parsing, and grammatical analysis.
Finally, block is a database.
If someone like, it can be easily used as an engine for fortran IDEs
(e.g. as a plugin of emacs, Eric4 or so, I guess).

>Regarding issue 8, point 1:
 OK. I see. No problem. 
 # for fpp can be included in fpp version of fparser in future.
 (As I have said, very limited function of fpp covers most of all fortran code.)

>How long it typically takes to parse your Fortran codes? 
I did
 /fatest.py a.F >fatest.a 2>fatest.a.err
with tree.torepr(500). a.F is 1.4Mbyte fortran source code by core-i7 2.93GHZ
python2.6. No problem.

>Regarding  issue 4 , point 1 and 2:
>What is the point of reproducing literally the original code?... 
I see. I imagine some program which rewrite my fortran codes; there are
possibilities. An example:
Information of subroutines is documented and embedded as comment.
  1. input arguments, 
  2. index arguments (special input which is used for array size), 
  3. output arguments
  4. input/output arguments.
  5. used materials in used module, used arguments in type. 
  6. used files, 
and so on. (these can be combined by mediawiki/extension so that just writing 
fortran
source in mediawiki edit-windows results in processed results). But this is 
just an
example. Careful design is required, and I need to think more.
Anyway, for such a purpose, it is meaningful to preserve original as it is. 
But I am not so sure; it is a cost/performance problem.
If it affects so much on fparser and related things; maybe useless.

With your minimum reply to this comment, you may close this issue. It would be 
better
to continue discussion in other place. Group or Issue?

Original comment by TakaoKot...@gmail.com on 23 Mar 2010 at 2:46

[GoogleCodeExporter]

I think bugs and feature requests should go to Issues.
Groups could be used to discuss the design and 
applications of fparser. Though, if group discussion
converges then anything related to fparser code such
as implementation, modifications, etc should again
end up in Issues.

Here's an idea about comments:
Rule 1: Every comment that follows a blank line
is interpreted as a documentation of a code
block following it.
Rule 2: Every comment that follows a Fortran language element
is interpreted as a documentation of the element.
Rule 3: Every comment that follows a comment is 
interpreted as a continuation of previous documentation.

For example:
----------------------
  subroutine foo(
 &a, ! a doc
! a doc continues
 &b ! b doc
  )
! foo doc
! foo doc continues
  real a, ! a doc
 &b, ! b doc..
     ! b doc continues
! b doc still continues

! This not b doc because of blank line.
! Instead, this can be interpreted as
! a documentation for what follows below.
  int i
  end subroutine
-----------------------

Let's discuss this idea in a separate issue
by coping the above idea if you think it would
be worth it.

Original comment by pearu.peterson on 23 Mar 2010 at 8:57

[GoogleCodeExporter]

Hi, thanks to your quick fixing. It encourages me so much. 
I have found I can scan one of all codes of my own fortran package.

>I think bugs and feature requests should go to Issues.
>Groups could be used to discuss the design and 
>applications of fparser. Though, if group discussion
>converges then anything related to fparser code such
>as implementation, modifications, etc should again
>end up in Issues.
OK. I will follow what you say. I should respect you a leader here. let me know 
when
you feel I put some article on wrong place.

>Here's an idea about comments:
>Rule 1: Every comment that follows a blank line
>...
Is this how to recover CommentBlock?

Original comment by TakaoKot...@gmail.com on 23 Mar 2010 at 8:09

[GoogleCodeExporter]

I have just committed a big non-trivial patch to svn that
fixes a bug with lines containing label and construct-name at the
same time. Could you test this against your fortran codes, is
the scan still successful for you?

I am closing this issue as most of the problems in the subject are fixed.
Please report any bugs to new issues.

No, I did not plan to recover the CommentBlock.
The idea was to attach comments to Fortran language items
in cases where comments can be interpreted as documentation
strings of these language items. This would enable recovering
the code
------------
     real a, ! a doc string
    & b ! b doc string
------------
as
------------
     REAL a, ! a doc string
    &b ! b doc string
------------
instead of the current behaviour:
-------------
     REAL a, b
     ! a doc string
     ! b doc string
-------------

Original comment by pearu.peterson on 23 Mar 2010 at 10:59

• Changed state: Fixed

[GoogleCodeExporter]

I think it is not so easy to identify a comment line belongs to which line.
Some people may write codes like this.
      program test
      ! a is for something
      real a, 
     &     b    
      end
So, key point is that "we need an easy way to reproduce the original source 
code as
it is". This is because I want to have a program which is to rewrite fortran 
codes in
future.(e.g. it will be possible to add intent to all variables. Further, embed
comment lines which shows that what component of instance in a type are 
modified or
read.)

In such a case, we take diff, and check what are done by the program.

Original comment by TakaoKot...@gmail.com on 24 Mar 2010 at 10:04