unnecessary re-compilation

[perrette]

Pretty amazed by the magic of FoBiS (and still checking it out). I see in the list of Main Features:

• avoid unnecessary re-compilation (algorithm based on file-timestamp value);

But from the output and the execution time, it feels like everything is recompiled every time systematically. For large programs this can be annoying to wait for many seconds everytime a "print" statement is inserted in the middle of the debugging process. I guess part of the time comes from re-parsing the dependency tree (by the way, I am sure there is a way of storing that tree somewhere for re-use?). But from the printed message, there is no difference between first and second compile.

My simple project:

• mymod.f90:

    module mymod
    contains
      subroutine print_hello()
        print *, "hello"
      end subroutine
    end module

• myprog.f90:

    program main
      use mymod
      call print_hello()
    end program

Compilation and log, whether the object/executable are already up-to-date or not:

FoBiS.py build -compiler gnu

Builder options
  Directories
    Building directory: "."
    Compiled-objects .o   directory: "obj"
    Compiled-objects .mod directory: "mod"
  Compiler options
    Vendor: "gnu"
    Compiler command: "gfortran"
    Module directory switch: "-J"
    Compiling flags: "-c"
    Linking flags: ""
    Preprocessing flags: ""
    Coverage: False
    Profile: False
  PreForM.py used: False
  PreForM.py output directory: None
  PreForM.py extensions processed: []

Building ./myprog.f90
Compiling ./mymod.f90 serially
Compiling ./myprog.f90 serially
Linking ./myprog
Target ./myprog.f90 has been successfully built

Am I just missing something, or is there a bug?

[perrette]

(I am using the pip version FoBiS.py 1.6.8 on Ubuntu LTS 14.04)

[szaghi]

Dear @perrette,

thank you for testing FoBiS.py.

The recompilation should be avoided, however I checked this feature in a not relatively recent version (maybe 1.0.x). It is possible that I have introduced some bugs in recent versions (I am thinking to the cflags-hystory feature...) that have broken the avoid-unnecessary-compilation check.

Besides the presence of this eventual bug, it is right that:

• each time FoBiS.py is runned the compilation hierarchy is recomputed; this is not a bug, it is my choice: FoBiS.py should be able to blind build all... especially into a "testing environment" where you change (add/remove) sources continuously;
• the hierarchy computation should not be so heavy in terms of computational cost, but it surely be optimized; please open a new issue asking for a more efficient compilation-hierarchy computation thus I will remember to try to improve it;
• the printing message must be changed: for object that are already uptodate the message must be different with respect the ones must be recompiled.

Stay connected, I will try to fix asap.

Thank you very much!

[perrette]

Thanks. That is fair. My suggestion with respect to recomputing the compilation hierarchy (but I suspect it is already there, have not checked all options yet) is to be able to save it to a file, and read it in. For large projects this might make a difference. If not yet implemented I can open an issue. But first I would like to check whether execution time is due to re-compilation or compilation hierarchy computation.

[szaghi]

Fixed. I have introduced a subtle bug when I purged out manually inserted os.sep substituting it with a more sage os.path.join. New version fixing this is 1.6.9 on PyPi (very soon on GitHub).

[szaghi]

Please, check if the new version obtain the performance you expect in recompilation. Moreover, can you try the multi-threads speedup if you a multicore WorkStation.

[perrette]

Thanks!

It is much better indeed:

• first compilation (or -force_compile activated): 5.078s
• subsequent compilation (actually only parsing/linking): 0.337s

[perrette]

P.S: being able to save the dependency tree somewhere might still be useful (speed-up, graph calculation etc...). It does not even need to be stable (back-compatible with every new release) since it can be re-generated any time. Hmmm, but to reduce maintenance and the number of options, as far as compilation time is concerned, really the simplest would be to get the Makefile generation right (a very useful feature, I am convinced).

[szaghi]

@perrette

You are absolutely right about the usefulness of makefile generation. What I previously intended is that I have other bugs-to-fix/feature-to-implement before fix the makefile generation, but it is planned to do (maybe before the next week).

As the saving of the tree hierarchy to a file I am confused: I cannot see how it can speedup the building, because I want FoBiS.py to always check the hierarchy in case files are added/removed/modified (FoBiS.py was born for an on-the-fly not-boring make replacement). A possible useful scenario could be if the user explicitly a flag for non recomputing the tree rather use the one previously saved. What do you think?

[perrette]

That is exactly what I meant To make the testing/development as smooth as possible, have a very fast re-compiling of the necessary files, without any non-necessary step, would be useful. But if the makefile generation works fine, then it is enough: just create the makefile, and for quick testing, just do a "make" every time a small modif is done. This is anyway faster to type than the full FoBiS.py command.

[szaghi]

@perrette

Just stop using makefile for a simple fobos file :-)

Seriously, I will try to give more support on makefile generation.

[perrette]

This looks like a powerful format to combine bunches of settings based on various modes, compilers etc... congrats !

It does not save the graph of dependencies though. The user has to rely on FoBiS internals (and trust back-compatibility) to be sure the exact same dependencies will be analyzed in future releases of FoBiS (who knows which version will be installed on the user's computer who has my code). That is where a basic Makefile reassures everyone (I definitely want to have a Makefile git-tracked in my project !)

One minor, low-priority suggestion for the fobos format:

• directories= parameter : in addition to src and exclude, have an optional directories (with similar functionality as python distutils' packages, but as : "dir1", "dir1/dir2", etc...) : increase user control AND clarity of what the program will be doing simply by looking at the fobos file. This is a tradeoff (IMO, fair) between full-automation and possible surprises (current FoBiS, despite all the good) and precise (over?)-determination (Makefile). I guess I am used to (and trust) python.

And in general, something easy to keep track of things: give FoBiS.py the option to generate the fobos file as well as the generated dependency graph, and to re-use them! The first would help writing templates / more complete fobos, the second would increase user control as well as compile time. Both would help ascertain what FoBiS.py does.

But again, from all these suggestions (since all they are concerned with is to increase reliability of FoBiS.py), have a correct Makefile that closely reflects FoBiS.py behaviour sounds the most important step... Ok, I stop with that ;-)

To temper my push toward Makefiles: your whole project is really impressive, so thanks and keep going !

[szaghi]

@perrette

This looks like a powerful format to combine bunches of settings based on various modes, compilers etc... congrats !

You are too kind!

It does not save the graph of dependencies though.

Yes, the FoBiS.py constructed dependency is into the log (that is created if you pass --log) not into the fobos, but it is a choice: I hate to write makefile for big project and one thing boring is exactly the creation of the compilation hierarchy rules.

directories= parameter : in addition to src and exclude...

Can you give me more details? I do not see the point... which information should contain directories?

give FoBiS.py the option to generate the fobos

Great idea! Please open a feature request on github issue!

have a correct Makefile that closely reflects FoBiS.py behaviour...

You trust more than in Make than in FoBiS.py... you are right! But, you are asking to FoBiS.py to create your makefile, you should not trust to those makefile :-)

Do not worry: improving the makefile generation in now my top list... today I will try to close the issue on handling non-module-library also into the generated makefile. Then I will search for makefile testers...

Thank you again for your help!

[perrette]

Cool!

It does not save the graph of dependencies though.

Yes, the FoBiS.py constructed dependency is into the log (that is created if you pass --log) not into the fobos, but it is a choice: I hate to write makefile for big project and one thing boring is exactly the >creation of the compilation hierarchy rules.

Sure, this is always this balance about control (takes time, cumbersome) and automation (easy, but no control in case of mistake). I was simply suggesting to make it possible for the user to have more control, in case something goes wrong - and at least, as a psychological tranquilizer when adopting a new tool (that latter reason also contributing to me insisting on generating a reliable Makefile from FoBiS.py). I keep thinking that a more deterministic mode (providing the exclusive list of files entering the project) would not be a bad option (or alternatively, re-read the log file - not yet checked what is inside).

My other suggestion of an optional directories= parameter goes in the same direction (enhance user's control). I mean simply, indicate the directories to be searched non-recursively for source files (note that most unix command do not attempt recursive search - probably because it is a dangerous feature - unless you explicitly ask for it with -r). I do not know whether you ever used python's distutils and its setup.py file, but there if you actually install a package (several files under a directory) instead of just a module (a single file.py), you are required to indicate which folders to consider as part of the package. So taking your "cumbersone" example from the README, the option would be:

directories = ["src/", "src/nested-1", "src/nested-1/nested-2"] 

The number of folders do not scale up as quickly as the number of files, and such a list is already quite informative when just having a look at it, regarding what is in the project, where the source files are etc...Besides, you can easily copy a folder from src_proj1 to src_proj1.old and keep editing src_proj1, without triggering an error, and without having to change your build script with a --exclude flag.

Maybe at some point you will even want to make such flags compulsory. Think of the "implicit none;" in fortran... What looks convenient at first may be judged error-prone later...

[szaghi]

Ok, finally I see your point. Now I understand the directory suggestion.

My comments:

• the building in FoBiS is not implicit; on the contrary it rely on the depency cascade that the programmer explicitely define... by means use module...; the magic (forgive me...) of FoBiS is that the programmer has not obligated to define two times (one in the fortran sources and the other into the makefile) this dependency;
• the above is not true for old fashioned, non-module-library with a blob of procedures; in this case there is not a dependency neither it can easy argued if the library has been actually used (I will need to parse sources for also procedures invocations... it is in my plan but not in the next months); as a consequence, these sources are compiled and passed to linker, but this is the same thing that it is commonly done with makefile by the (ab)use of wildcards and implicit compiling rules (these are really implicit);

Pointed out the above comments, my feel is that you are right: it can be desiderable (in some scenario) to give the user more control (other than the exclude option). I am thinking to a new option like only = ['dir1','dir1/dir2','dir3'...] where the user specify the only dirs to be considered with recursive search disable. If you like this, please open a new issue with a feature request (not exact this, but not too far I hope...)

Thank you again, see you soon

[jacobwilliams]

I also like the option of alternately specifying the directories. But, don't get rid of the current behavior, which is also great.

Make, on the other hand, is terrible, and should never be used. :)

[zbeekman]

Why are you all hating on make so much? :trollface:

Well actually I know… remind me to finish and upload my semi-sane fortran flymake repository sometime… It uses some painful Makefile wizardry to determine Fortran dependencies and interface with Emacs’ flymake-mode. It’s real great when flymake is working properly, but man it is/was a royal pain to get somewhat functional.

[jacobwilliams]

Emacs is also terrible.

Drops mic

[zbeekman]

Shots fired!! :gun:

what do you use? visual studio doesn’t count…

[jacobwilliams]

Ha ha! I'm just trolling you. I do love my Visual Studio, though!

On my Mac, all I need is TextWrangler. :)

[zbeekman]

I was actually looking into other good Fortran IDEs (for my coworker) and almost everything I’ve looked at seems kludgy or a super steep learning curve. Theoretically it shouldn’t be hard to find something decent that is a good/sane Fortran config out of the box, but I guess I’m just too optimisitic. I’ve looked into:

• Code::Blocks
• Visual Studio Code with @ tomedunn’s vs.language.fortran
• Photran/eclipse

Any way, my semi-sane flymake config and crazy emacs stuff works for me. Here’s an action shot in hopes of making you jealous… seeing syntax errors in near real time is nice.

[zbeekman]

just started messing around with the text wrangler… it seems… austere

[szaghi]

No no no.... you are all wrong!

Fortran poor people use... VIM

It is not only the best editor, it is best IDE killer :-)

https://github.com/szaghi/dotfiles

Only one thing is currently missing: a vim-plugin for FoBiS.py :-) indeed I use dispatc for running FoBis ....wonderfull!

[Tobychev]

Yeah @szaghi is correct: Vim rules!!!!

[perrette]

@szaghi The main advantage of a directories= parameter would be to ensure that no recursive search is made (with risk of infinite loop - symbolic links, or too long searching/parsing time for a deep folder tree containing code-unrelated files).

@jacobwilliams I would suggest not to get rid of the current behavior, but to make it more explicit by forcing the use of a -r or --recursive option. If none of -r or --directories is specified, only files in the current folder are searched for a parsed.

I will open an issue.

[szaghi]

@perrette

I prefer the other choice: the recursive remains the default behaviour while specifying directories = (or only =) the recursion will be disable and source searched only where the user specify.

Anyhow, please open a new issue where we can discuss.

Thank you very much for your help.