szaghi / FoBiS

FoBiS.py, Fortran projects Building System for poor people
138 stars 35 forks source link

unnecessary re-compilation #61

Closed perrette closed 9 years ago

perrette commented 9 years ago

Pretty amazed by the magic of FoBiS (and still checking it out). I see in the list of Main Features:

But from the output and the execution time, it feels like everything is recompiled every time systematically. For large programs this can be annoying to wait for many seconds everytime a "print" statement is inserted in the middle of the debugging process. I guess part of the time comes from re-parsing the dependency tree (by the way, I am sure there is a way of storing that tree somewhere for re-use?). But from the printed message, there is no difference between first and second compile.

My simple project:

    module mymod
    contains
      subroutine print_hello()
        print *, "hello"
      end subroutine
    end module
    program main
      use mymod
      call print_hello()
    end program

Compilation and log, whether the object/executable are already up-to-date or not:

FoBiS.py build -compiler gnu

Builder options
  Directories
    Building directory: "."
    Compiled-objects .o   directory: "obj"
    Compiled-objects .mod directory: "mod"
  Compiler options
    Vendor: "gnu"
    Compiler command: "gfortran"
    Module directory switch: "-J"
    Compiling flags: "-c"
    Linking flags: ""
    Preprocessing flags: ""
    Coverage: False
    Profile: False
  PreForM.py used: False
  PreForM.py output directory: None
  PreForM.py extensions processed: []

Building ./myprog.f90
Compiling ./mymod.f90 serially
Compiling ./myprog.f90 serially
Linking ./myprog
Target ./myprog.f90 has been successfully built

Am I just missing something, or is there a bug?

perrette commented 9 years ago

(I am using the pip version FoBiS.py 1.6.8 on Ubuntu LTS 14.04)

szaghi commented 9 years ago

Dear @perrette,

thank you for testing FoBiS.py.

The recompilation should be avoided, however I checked this feature in a not relatively recent version (maybe 1.0.x). It is possible that I have introduced some bugs in recent versions (I am thinking to the cflags-hystory feature...) that have broken the avoid-unnecessary-compilation check.

Besides the presence of this eventual bug, it is right that:

Stay connected, I will try to fix asap.

Thank you very much!

perrette commented 9 years ago

Thanks. That is fair. My suggestion with respect to recomputing the compilation hierarchy (but I suspect it is already there, have not checked all options yet) is to be able to save it to a file, and read it in. For large projects this might make a difference. If not yet implemented I can open an issue. But first I would like to check whether execution time is due to re-compilation or compilation hierarchy computation.

szaghi commented 9 years ago

Fixed. I have introduced a subtle bug when I purged out manually inserted os.sep substituting it with a more sage os.path.join. New version fixing this is 1.6.9 on PyPi (very soon on GitHub).

szaghi commented 9 years ago

Please, check if the new version obtain the performance you expect in recompilation. Moreover, can you try the multi-threads speedup if you a multicore WorkStation.

perrette commented 9 years ago

Thanks!

It is much better indeed:

perrette commented 9 years ago

P.S: being able to save the dependency tree somewhere might still be useful (speed-up, graph calculation etc...). It does not even need to be stable (back-compatible with every new release) since it can be re-generated any time. Hmmm, but to reduce maintenance and the number of options, as far as compilation time is concerned, really the simplest would be to get the Makefile generation right (a very useful feature, I am convinced).

szaghi commented 9 years ago

@perrette

You are absolutely right about the usefulness of makefile generation. What I previously intended is that I have other bugs-to-fix/feature-to-implement before fix the makefile generation, but it is planned to do (maybe before the next week).

As the saving of the tree hierarchy to a file I am confused: I cannot see how it can speedup the building, because I want FoBiS.py to always check the hierarchy in case files are added/removed/modified (FoBiS.py was born for an on-the-fly not-boring make replacement). A possible useful scenario could be if the user explicitly a flag for non recomputing the tree rather use the one previously saved. What do you think?

perrette commented 9 years ago

That is exactly what I meant To make the testing/development as smooth as possible, have a very fast re-compiling of the necessary files, without any non-necessary step, would be useful. But if the makefile generation works fine, then it is enough: just create the makefile, and for quick testing, just do a "make" every time a small modif is done. This is anyway faster to type than the full FoBiS.py command.

szaghi commented 9 years ago

@perrette

Just stop using makefile for a simple fobos file :-)

Seriously, I will try to give more support on makefile generation.

perrette commented 9 years ago

This looks like a powerful format to combine bunches of settings based on various modes, compilers etc... congrats !

It does not save the graph of dependencies though. The user has to rely on FoBiS internals (and trust back-compatibility) to be sure the exact same dependencies will be analyzed in future releases of FoBiS (who knows which version will be installed on the user's computer who has my code). That is where a basic Makefile reassures everyone (I definitely want to have a Makefile git-tracked in my project !)

One minor, low-priority suggestion for the fobos format:

And in general, something easy to keep track of things: give FoBiS.py the option to generate the fobos file as well as the generated dependency graph, and to re-use them! The first would help writing templates / more complete fobos, the second would increase user control as well as compile time. Both would help ascertain what FoBiS.py does.

But again, from all these suggestions (since all they are concerned with is to increase reliability of FoBiS.py), have a correct Makefile that closely reflects FoBiS.py behaviour sounds the most important step... Ok, I stop with that ;-)

To temper my push toward Makefiles: your whole project is really impressive, so thanks and keep going !

szaghi commented 9 years ago

@perrette

This looks like a powerful format to combine bunches of settings based on various modes, compilers etc... congrats !

You are too kind!

It does not save the graph of dependencies though.

Yes, the FoBiS.py constructed dependency is into the log (that is created if you pass --log) not into the fobos, but it is a choice: I hate to write makefile for big project and one thing boring is exactly the creation of the compilation hierarchy rules.

directories= parameter : in addition to src and exclude...

Can you give me more details? I do not see the point... which information should contain directories?

give FoBiS.py the option to generate the fobos

Great idea! Please open a feature request on github issue!

have a correct Makefile that closely reflects FoBiS.py behaviour...

You trust more than in Make than in FoBiS.py... you are right! But, you are asking to FoBiS.py to create your makefile, you should not trust to those makefile :-)

Do not worry: improving the makefile generation in now my top list... today I will try to close the issue on handling non-module-library also into the generated makefile. Then I will search for makefile testers...

Thank you again for your help!

perrette commented 9 years ago

Cool!

It does not save the graph of dependencies though.

Yes, the FoBiS.py constructed dependency is into the log (that is created if you pass --log) not into the fobos, but it is a choice: I hate to write makefile for big project and one thing boring is exactly the >creation of the compilation hierarchy rules.

Sure, this is always this balance about control (takes time, cumbersome) and automation (easy, but no control in case of mistake). I was simply suggesting to make it possible for the user to have more control, in case something goes wrong - and at least, as a psychological tranquilizer when adopting a new tool (that latter reason also contributing to me insisting on generating a reliable Makefile from FoBiS.py). I keep thinking that a more deterministic mode (providing the exclusive list of files entering the project) would not be a bad option (or alternatively, re-read the log file - not yet checked what is inside).

My other suggestion of an optional directories= parameter goes in the same direction (enhance user's control). I mean simply, indicate the directories to be searched non-recursively for source files (note that most unix command do not attempt recursive search - probably because it is a dangerous feature - unless you explicitly ask for it with -r). I do not know whether you ever used python's distutils and its setup.py file, but there if you actually install a package (several files under a directory) instead of just a module (a single file.py), you are required to indicate which folders to consider as part of the package. So taking your "cumbersone" example from the README, the option would be:

directories = ["src/", "src/nested-1", "src/nested-1/nested-2"] 

The number of folders do not scale up as quickly as the number of files, and such a list is already quite informative when just having a look at it, regarding what is in the project, where the source files are etc...Besides, you can easily copy a folder from src_proj1 to src_proj1.old and keep editing src_proj1, without triggering an error, and without having to change your build script with a --exclude flag.

Maybe at some point you will even want to make such flags compulsory. Think of the "implicit none;" in fortran... What looks convenient at first may be judged error-prone later...

szaghi commented 9 years ago

Ok, finally I see your point. Now I understand the directory suggestion.

My comments:

Pointed out the above comments, my feel is that you are right: it can be desiderable (in some scenario) to give the user more control (other than the exclude option). I am thinking to a new option like only = ['dir1','dir1/dir2','dir3'...] where the user specify the only dirs to be considered with recursive search disable. If you like this, please open a new issue with a feature request (not exact this, but not too far I hope...)

Thank you again, see you soon

jacobwilliams commented 9 years ago

I also like the option of alternately specifying the directories. But, don't get rid of the current behavior, which is also great.

Make, on the other hand, is terrible, and should never be used. :)

zbeekman commented 9 years ago

Why are you all hating on make so much? :trollface:

Well actually I know… remind me to finish and upload my semi-sane fortran flymake repository sometime… It uses some painful Makefile wizardry to determine Fortran dependencies and interface with Emacs’ flymake-mode. It’s real great when flymake is working properly, but man it is/was a royal pain to get somewhat functional.

jacobwilliams commented 9 years ago

Emacs is also terrible.

Drops mic

zbeekman commented 9 years ago

Shots fired!! :gun:

what do you use? visual studio doesn’t count…

jacobwilliams commented 9 years ago

Ha ha! I'm just trolling you. I do love my Visual Studio, though!

On my Mac, all I need is TextWrangler. :)

zbeekman commented 9 years ago

I was actually looking into other good Fortran IDEs (for my coworker) and almost everything I’ve looked at seems kludgy or a super steep learning curve. Theoretically it shouldn’t be hard to find something decent that is a good/sane Fortran config out of the box, but I guess I’m just too optimisitic. I’ve looked into:

Any way, my semi-sane flymake config and crazy emacs stuff works for me. Here’s an action shot in hopes of making you jealous… seeing syntax errors in near real time is nice.

fortran-flymake

zbeekman commented 9 years ago

just started messing around with the text wrangler… it seems… austere

szaghi commented 9 years ago

No no no.... you are all wrong!

Fortran poor people use... VIM

It is not only the best editor, it is best IDE killer :-)

https://github.com/szaghi/dotfiles

Only one thing is currently missing: a vim-plugin for FoBiS.py :-) indeed I use dispatc for running FoBis ....wonderfull!

Tobychev commented 9 years ago

Yeah @szaghi is correct: Vim rules!!!!

perrette commented 9 years ago

@szaghi The main advantage of a directories= parameter would be to ensure that no recursive search is made (with risk of infinite loop - symbolic links, or too long searching/parsing time for a deep folder tree containing code-unrelated files).

@jacobwilliams I would suggest not to get rid of the current behavior, but to make it more explicit by forcing the use of a -r or --recursive option. If none of -r or --directories is specified, only files in the current folder are searched for a parsed.

I will open an issue.

szaghi commented 9 years ago

@perrette

I prefer the other choice: the recursive remains the default behaviour while specifying directories = (or only =) the recursion will be disable and source searched only where the user specify.

Anyhow, please open a new issue where we can discuss.

Thank you very much for your help.