Support for fypp preprocessor

[mobius-eng]

Fortran's preprocessor is rather weak. fypp is a good alternative and it is used by quite a few projects. So, it would be nice if the linter could support it.

I don't know if this at all feasible though... Basically, it would require running fypp before compiling for linting. But then, of course, the lines of errors all become all over the place in the original file.

[aradi]

Just a short note on this. When fypp is run with the -n option for generating line markers, the gfortran error messages will contain the correct line numbers (the ones referring to the unpreprocessed file).

[gnikit]

From what I understand when using fypp, one edits the .fpp file and the proceeds to run fypp to generate the the .f90 files. If using the option -n yields to a gfortran compilamble piece of code, which as far as I could test it does, this should not be so hard to implement since you could parse the output of fypp into gfortran.

As long as as each .fpp file is self-contained i.e. you can always do fypp ${args} file_in file_out this should be relatively easy to do. If you need to include more that one source files into fypp to get the compilable output for gfortran, then I am afraid it won't be easily doable.

For the linter to work I suspect that all the preprocessor variables and include paths will have to be passed in fortran.linterExtraArgs. My question to you is would it be safe to make this assumption? i.e. ALL -I and -D arguments passed in fypp should also be passed in gfortran as well?

[aradi]

The -I and -D options for fypp and gfortran need not to be identical. The -I option would tell fypp where to look up fypp include files (included via the #:include option), while gfortran would use it to look up .mod files. For -D, I guess, fypp would evaluate all the variables in the source code, so gfortran would not need to expand/substitute anything. (Unless, you mix fypp-styled and cpp-styled variables within the source code, which is probably a questionable practice). I think, it would be safe to pass all options to both, fypp and gfortran, if that is technically easier to realize. (E.g. -I could be specified for all fypp-include directories and all directories with .mod files and passed to both programs.)

[gnikit]

I actually had a play yesterday and it's trivial to have different -I flags for fypp and gfortran; I think I overcomplicated things in my head. It is slightly annoying since we essentially will need to create a wrapper around things like the linter, the hover provider, etc.

I will try and get around doing this this week depending on how PhD workload turns out. The current solution I had in mind is creating a .F90 file when we lint and then use that. There are many cases where this would be problematic but capturing the output in stdout and parsing it around would require most parts of the extension to be rewritten. Probably what we can do is use some other extension e.g. .fpptmp so built tools that use globs can still work.

If you have any examples of source files for fypp that we could use to test the robustness of the fypp linter that would be great, otherwise we can always copy the examples from the fypp repo.

The main challenge I have identified with adding fypp support is that almost everywhere in the extension we use vscode.TextDocument to get the filename and other document properties. If we add fypp in the mix then vscode.TextDocument will not refer to the correct document

I am scribbling down a few comments that I will move to the PR eventually.

Initial TODO List

• [ ] Generate a call to fypp before gfortran to get the Fortran source file
• [x] Edit syntax highlighting to handle #:
• [ ] If fypp replace the vscode.TextDocument file extension with custom extension. This is a horrible, unsafe idea but it should work for now since the Linter is the first thing the extension calls so everyone else should have access to the generated source file. NOTE: that the new formatting features being added in #256 might not be working with fypp

[aradi]

You can take fypp-files from the Fortran Standard Library as example, or from DFTB+ (which was actually the reason, why Fypp had been created :wink:)

By the way, fprettify has Fypp-support.

[jphaupt]

Any news, or at least workarounds on this? It would be nice to at least have some syntax highlighting/not get spurious errors.

[gnikit]

I think you should have syntax highlighting already. The actual implementation is a bit more tricky and between this project, fortls and my PhD I am a bit low on time.

[jphaupt]

Thanks, that's completely understandable. Sorry, I only just noticed it seems I have syntax highlighting, but only for certain file extensions. It doesn't seem to like .fpp files (.fypp and .fpph are fine). Anyway, thanks for the work on this; making the start of my own PhD a lot smoother. :D

[gnikit]

It doesn't seem to like .fpp files

You can tell VSCode to associate a file extension with a certain syntax by Ctrl + K + M and selecting Fortran90

start of my own PhD a lot smoother

Glad to hear that!

[jphaupt]

It is associated with Fortran90 and some syntax highlighting is there, but it does not recognise #:.

[gnikit]

Could you post a minimal example of the code failing to highlight?

[jphaupt]

Sorry, found myself in a meeting. Here's one I stole from the fortran wiki: if I name it (e.g.) test.fypp the highlighting is fine, but not if I call it test.fpp.

! minimal example for syntax highlighting

#:if DEBUG > 0
  print *, "Some debug information"
#:endif

#:set LOGLEVEL = 2

#:del LOGLEVEL

[gnikit]

By highlighting do you mean the red error swigles at the bottom? Because for me both look the same

[jphaupt]

You are right, I did not use the right terminology. I meant that it gives errors. Anyway, I do actually have an example of a case where syntax highlighting doesn't work, but it doesn't work with either filename extension (but as before only fpp shows error squiggles):

#:set data_names = [ 'eg1', 'eg2' ]
module min_eg
  implicit none
  private

#:for dname in data_names

  public :: eg_${dname}$_t
  type :: eg_${dname}$_t
  contains
    procedure :: do_something => do_something_${dname}$
  end type eg_${dname}$_t

#:endfor

contains

#:for dname in data_names
  subroutine do_something_${dname}$ (this)
    implicit none
    class(eg_${dname}$_t) :: this
    print*, "this subroutine has no syntax highlighting with either extension"
  end subroutine do_something_${dname}$
#:endfor

end module min_eg

The subroutine does not have syntax highlighting, but it is recovered if you remove eg_${dname}$_t in end type eg_${dname}$_t. (I haven't properly checked if this is valid code, but it is a mowed down version of a file with which I am actually working).

[gnikit]

I did not use the right terminology. I meant that it gives errors

That is fine. Those errors are from the linter, which is what this issue is about.

As for the syntax highlighting, we will do our best so support fypp highlighting but given that it can have pretty complex structures, from a highlighting point of view, e.g. logical, parameter :: hasMpi = #{if defined('MPI')}#.true.#{else}#.false.#{endif}# I don't think we will ever be able to have full-blown fypp highlighting. Unless of course any of the fypp devs, which actually know fypp in depth, help us write the TextMate files.

[aradi]

@gnikit I am one of the fypp devs, and definitely willing to help. However, I've never created TexMate files or anything alike so far, so you would probably have to guide me and let me know, what you exactly need. :wink:

[gnikit]

@gnikit I am one of the fypp devs, and definitely willing to help. However, I've never created TexMate files or anything alike so far, so you would probably have to guide me and let me know, what you exactly need. wink

Okay, that is awesome @aradi. I will try and make a start after I merge version 3.x into master. I will tag you to the PR then!

[gnikit]

Hi @aradi sorry this has taken longer than expected (have been busy with PhD write up). I was wondering if you have any thoughts on how to implement this, specifically I would be curious to know if you have any ideas on how to pass the fypp output to the compiler. The way I see it there are 2 possible ways we can go:

1. Run fypp -> capture output in an Output stream -> pass that Output to gfortran/ifort/... -> get compiler diagnostics ->map from expanded source file back to fypp format.
2. Run fypp ->save to a file (possibly temp?) ->pass (possibly temp) file to gfortran/ifort/... -> get compiler diagnostics -> map from expanded source file back to fypp format (-> delete temp file?).

I would like to implement 1. but I don't think that it's supported for Fortran in GNU and same goes for Intel AFAIK. That means that we have to use 2. My concerns with the latter approach are:

• Where do you save the files? They need to be in a a location which by default can reach all the needed .mod files and the directory structure of where you are placing the temp files needs to mimic that of src.
• How do you name the files to avoid conflicts?
• What about performance? Running fypp + the compiler + creating a file + all the other steps everytime you save will come at a cost. Will it be noticeable?

@awvwgk I think you had some thoughts on this during June's Monthly call.

In case you want to continue this conversation in private, feel free to reach out to me on Fortran Discourse @gnikit

[aradi]

@gnikit No problem, I hope, you were successful in writing up your thesis. :smile:

A few notes:

• I am definitely in favor of avoiding intermediate files, whenever possible.
• GFortran seems to be able to process source on stdin:

  echo -e "print *, 'hello'\nend\n" | gfortran -x f95 -

• Independent of the source handling, one should probably anyway consider to set the -J option for gfortran, in order to collect all modfiles at one place in a clever way. I am not sure, what the right strategy is, I just find somewhat disturbing, when .mod and .smod files pop up in the source directory. (But this is clearly not an fypp related issue)