Fortran 2008: SCons don't recognize .smod files

[jglezplatas]

Version of Scons: 3.0.5

Version Python: 2.7.10

Python distribution: brew

Installation Scons: pip install scons

Plataform: Any

How reproduce the problem: In fortran 2008 is possible to define submodules and in this case, the compiler create additional *.smod files that scons don't recognize. It has some consequences as example, when you want to execute scons -c to clean all. In this case delete all except .smod files. Example.zip

The command line to execute scons:

scons --tree=prune 
scons -c

[mwichmann]

I can see that the scons Fortran tool indeed has no knowledge of the submodule files. Support for this aspect of the 2008 standard seems to be quite recent - in the Gnu Compiler Collection I see it requires 6.0 as a minimum version, as noted here: https://gcc.gnu.org/wiki/Fortran2008Status

This ISO Technical Report provides details on the feature.

The Fortran tool looks like it would need work in both the emitter and scanner. It may be as simple as just editing the regex, or not - note that GNU Fortran at least has some exceptions:

Submodules are supported. It should noted that MODULEs do not produce the smod file needed by the descendent SUBMODULEs unless they contain at least one MODULE PROCEDURE interface. The reason for this is that SUBMODULEs are useless without MODULE PROCEDUREs. See http://j3-fortran.org/doc/meeting/207/15-209.txt for a discussion and a draft interpretation. Adopting this interpretation has the advantage that code that does not use submodules does not generate smod files.

[jglezplatas]

It was only as example...You have also the define directly submodules files...

[jglezplatas]

For me are different things. One problem is that Scons don't recognize the smod files and other thing is the order compilation in windows. Anything is different from Linux that in Windows?

[bdbaddog]

They are likely not different things. If SCons is aware of the .smod files, then it would be able to compile in the proper order right?

[jglezplatas]

I think that the problem isn't with smod files, it is related about the information on directive use (access to others modules) that there are in fortran files. Therefore if a module (called B for example) is "using" other module (called A), it have to be created compiling the module (A) before than (B). "Logical process" ...

[bdbaddog]

As an expert on SCons.. you are mistaken.
In general guessing how software works based on an incomplete understanding is almost always wrong.

Once SCons knows about smod files, and knows about dependencies on smod files, then the build order will be correct and both this and your windows order issue will be resolved.

[jglezplatas]

Therefore that scons compile and link in MacOS the same example that in Windows without problems, means that it was "casual". OK. Understood.

[bdbaddog]

I'm not sure what you mean by casual.

In general arguing with the subject matter experts who are trying to help you is not fruitful.

[jglezplatas]

Completely agree with you. This will be my last message. Thanks for all

[mwichmann]

If we could get a statement like the three following for the SUBMODULE case, possibly someone who doesn't know Fortran could still produce the regex...

#   The USE statement regex matches the following:
#
#   USE module_name
#   USE :: module_name
#   USE, INTRINSIC :: module_name
#   USE, NON_INTRINSIC :: module_name

#   The INCLUDE statement regex matches the following:
#
#   INCLUDE 'some_Text'
#   INCLUDE "some_Text"
#   INCLUDE "some_Text" ; INCLUDE "some_Text"
#   INCLUDE kind_"some_Text"
#   INCLUDE kind_'some_Text"
#
#   where some_Text can include any alphanumeric and/or special character
#   as defined by the Fortran 2003 standard.

#   The MODULE statement regex finds module definitions by matching
#   the following:
#
#   MODULE module_name
#
#   but *not* the following:
#
#   MODULE PROCEDURE procedure_name

[pdiener]

As I have recently seen some issues with building in parallel that I expect to be related to this issue, I decided to look into how different compilers handles submodules and I will try to summarize my understanding of this here. I looked at gnu's gfortran, intel's ifort and pgi's pgfortran.

Submodules are intended to be used to separate the interface of a module with the implementation, in order to, among other things, avoid recompile cascades if the implementation of a module changes but the interface does not. Any fortran routine that USE a module does not need to be recompiled if the interface does not change.

Submodules are fairly different from modules. The main difference is that other fortran code will never USE a submodule. In the case of just 1 levels of submodules, any submodule will have one (and only one) parent module. If there are multiple levels of submodules, the submodules parent will be another submodule, but then it will still have an ancestor module. Submodules may USE other modules and have INCLUDE statements. There may be multiple modules and submodules defined in the same fortran file.

A first level submodule is defined by:

SUBMODULE(module_name) submodule_name

In this case the module module_name is the parent of the submodule submodule_name.

Submodules can also have submodules, in which case the definition is:

SUBMODULE(module_name:submodule_name) submodule_name2.

Here the submodule submodule_name2 is defined with the submodule submodule_name as it's parent and the module module_name as it's ultimate ancestor.

The definition of a third level submodule can then be done with:

SUBMODULE(module_name:submodule_name2) submodule_name3,

which defines submodule submodule_name3 to have the submodule submodule_name2 as it's parent and again module module_name as it's ultimate ancestor.

Unfortunately I found that the three compilers I looked at (gnu, intel and pgi) all handles submodules slightly differently. Whenever a fortran file with a module is compiled, all three compilers generate a .mod file named after the module name, i.e. module_name.mod. The gnu compiler will, in addition, generate a '.smod' file named after the module, i.e. module_name.smod. This happens even if the file only contains the module definition. I'm not sure what criteria needs to be satisified for the compiler to do this. In all the cases I have checked, the .mod and .smod files are identical.

When a file containing one or more first level submodules is compiled both gnu and intel compilers will generate a .smod file for each submodule. The name of that file is module_name@submodule_name.smod. The pgi compiler on the other hand never produces .smod files, instead it will produce an additional .mod files for each submodule. The name of that file is module_name-submodule_name.mod. For the intel and pgi compilers compilation of the file with the submodule requires the presence of the .mod file of the parent module, whereas the gnu compiler requires the presence of the '.smod' file of the parent.

For deeper levels of submodules the behavior is similar. The gnu and intel compilers produces .smod files for each submodule with the same naming scheme as above, i.e. for the case of a submodule of a submodule (as above) the name will be module_name@submodule_name2.smod. Similarly the pgi compiler produces .mod files with the name module_name-submodule_name2.mod. Here there is an additional slight difference. When using the gnu compuler only the .smod file if the immediate parent is required. However, when using the intel or pgi compilers all the .smod (for intel) and .mod (for pgi) files for all parent and ancestor submodules as well as the .mod file for the ancestor module are required.

As mentioned above submodules can USE other modules and also INCLUDE, so this of course adds additional dependencies.

The final link stage does not require any of the .mod or .smod files. Only the .o files produced from the files that contains submodules are needed at that point.

It is of course unfortunate that different compilers behave so differently, but as far as I know the Fortran standard only defines how modules should behave from a programmers standpoint and not how compilers work with them internally. I currently do now have access to any other Fortran compilers, so there could be some other possible behavior out there.

I hope this information is enough for somebody, more familiar with the SCons build system than I, to find a way to add proper submodule support to SCons. If something is still unclear, please ask and I'll be happy to investigate and try to clarify.

[nDimensionalSpace]

Just wondering if this issue has seen any progress. Maybe interestingly, on v3.1.2, a serial build (-j 1) works fine, but a parallel build fails . . . I haven't mucked around with scons source much yet, but as this is causing us a fair bit of pain, I will, if the ultimate answer is null, or "no progress".

[bdbaddog]

Please try latest SCons and verify this is still an issue? (most likely it is, but there were lots of changes in 4.0.0)

If not, PR's welcome. Please come to our discord server #devel if you want help on how to get started with such. Also see: https://github.com/SCons/scons/wiki/DeveloperGuide

[pdiener]

After I performed the investigation of how different compilers handled smod files, I unfortunately didn't have time to follow up and after a while didn't think about it anymore. Mainly because I don't see too many problems with this in my daily work. Thanks for pinging the issue. Maybe it's time for me to get on the the discord server and try to get things moving again.

[nDimensionalSpace]

Well, I am going to start working on it as well, and I figure its more likely to get done if we have two semi-distracted people working on it than just one semi-distracted person . . .

[bdbaddog]

@rsf83F4qW26jC6MVqBVC and @pdiener - no reason you couldn't work off a shared branch and do PR from your repo to the other.

[nDimensionalSpace]

I am happy to do that . . . I will let @pdiener set it up, if he likes.

FYI, I just ran the latest commit, and it behaves exactly as 3.1.2 did, as anticipated.

[bdbaddog]

Thanks for verifying. I checked the CHANGES.txt and that's usually all inclusive, but you never know sometimes extra functionality slips by me.

On Thu, Aug 13, 2020 at 2:55 PM rsf83F4qW26jC6MVqBVC < notifications@github.com> wrote:

I am happy to do that . . . I will let @pdiener https://github.com/pdiener set it up, if he likes.

FYI, I just ran the latest commit, and it behaves exactly as 3.1.2 did, as anticipated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SCons/scons/issues/3366#issuecomment-673729787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAUCNHR6Z4ZABLIKPZPK5TSAROO3ANCNFSM4HK5PRMA .

[pdiener]

@rsf83F4qW26jC6MVqBVC: You should have gotten an invite to the fork of scons I created.

[nDimensionalSpace]

Got it, thanks.

On Aug 14, 2020, at 9:28 AM, pdiener notifications@github.com wrote:

@rsf83F4qW26jC6MVqBVC: You should have gotten an invite to the fork of scons I created.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[nDimensionalSpace]

Thanks for verifying. I checked the CHANGES.txt and that's usually all inclusive, but you never know sometimes extra functionality slips by me.

No worries . . . I definitely don't expect everyone to be on top of everything at all times.

So, I just fixed up my test setup, and it turns out that even the serial case was only conditional . . . I didn't realize yesterday that non-dependant files in the same directory seem to get sorted into alphabetical order by filename. Now that I have renamed the files into a better order, it is failing consistently, even for -j 1.

I have looked at the code, and it seems mostly straightforward . . . I will still have a few questions (e.g., how to handle pgi, per @pdiener's comments above), but I will ask them over on discord.

[nDimensionalSpace]

@pdiener It looks like your repo is a little out of date . . . Care to pull the latest master branch? Otherwise, I will just submit PRs against the main repo here. Lmk, thx

[pdiener]

@rsf83F4qW26jC6MVqBVC Sorry, that apparently was my old fork I used last time I contributed to SCons. When I tried to update that, I got conflicts, so instead I deleted that old fork and reforked. I have sent you an invite to the new fork.