added cmake out of src workflow

JorySchossau commented 4 years ago

First part of a deprecation of mbuild.py project generation. The functionality has been replaced by a cmake out of source workflow: 1) git clone mabe (shorthand) 2) edit mabe/buildOptions.txt 3) mkdir build 4) cmake ../mabe -G where makefiles are default, and your system may support: Visual Studio, XCode, CodeBlocks, Ninja, and others.

Benefits:

Out of source workflow to not dirty the repo clone.
A more standard software practice that is no more difficult than we had before.
We don't have to support project generation code anymore.
buildOptions now allows adding custom cmake fragments as filenames, which opens the door to easily adding external libraries (ex: 3d graphics, fast linear algebra, etc.)

Note, for Visual Studio, you will need to be in a Visual Studio Environment command prompt (it's in the start menu somewhere - type Visual Studio)

Tested on Windows 10, OSX Catalina, HPCC, MSYS2/MinGW.

Why didn't we do this before? Because only in the last year did Visual Studio's cmake support become decent.

Shalmezad commented 4 years ago

Couple things to note for anyone trying this as well: 1) Full command is cmake ../MABE -G <generator>, replacing <generator> with your build option 2) Note case sensitivity does matter on some systems (doesn't matter on OS X by default, does matter on most linux systems) 3) May want to consider if it's possible to lower the minimum cmake required to 3.10

Shalmezad commented 4 years ago

Nevermind point 3. FindPython3 is only available in 3.12+ cmake.

cliff-bohm commented 4 years ago

Does mbuild -g still work? when will that be depricated? What is the plan for communicating this (i.e. will the -g option on mbuild provide the correct new command or link to usage?)

What happened to 24 h review?

JorySchossau commented 4 years ago

@Shalmezad Yep! I need to fix the version check as I think we need to support 3.12, but no older than that. And for caseness, 'MABE' is only capitalized if that's how you cloned it.

@cliff-bohm mbuild -g still works, but to get the makefile you have to explicitly ask for it now ('mk' or 'make'). The '--noCompile' option now is default (actually, it always was) EDIT: scratch that last sentence. I've only removed the behavior of strangely generating a makefile when 'not compiling'. We can change this, but really I want to delete that gigantic custom project generation code at some point anyway.

cliff-bohm commented 4 years ago

I just attempted a build:

$ python pythonTools/mbuild.py -p 10 make: *** No targets specified and no makefile found. Stop. Building MABE with:

World Empty

Genome Circular

Brain Human

Optimizer Simple

Archivist Default SSwD

JorySchossau commented 4 years ago

You now have to be explicit: make a makefile. Call the makefile: mbuild.py -g make && make -j10. The new workflow is cmake ../mabe which does the same thing as mbuild.py -g make, but better.

JorySchossau commented 4 years ago

If it's a pain I can make that easy again.

cliff-bohm commented 4 years ago

Okay, that command works. Cool.

but... 1) It does not maintain the old behavior so it really should have had a review period.

2) there is now some text at the bottom of build options, but it does not say anything along the lines of "If you are doing a normal build the old fashion way you can ignore this"

3) I think that the default behavior for

python pythonTools/mbuild.py

should be to generate an executable. This is an approachability issue for new users as must as anything else. A user's first experience with MABE is, pull, run mbuild, run mabe. It's nice and simple. If we want to have the code live in one directory and the executable get generated somewhere else I'm fine with that. If we are changing the layout of directories, I would prefer that we set it up such that we have:

/code
/make and object (i.e. where the build happens) this could be in /code of course
/work (where the executable gets placed)

and we run mbuild.py from the directory above. This way if I don't care about code or the make process I don't need to look at it. I just mbuild and go into work to get my executable. I also like this layout because it supports copying the work directory to create experiments.

That all being said... Can you spell out in one message the exact commands I need to run to see the new workflow. I'm still not getting it.

Shalmezad commented 4 years ago

Workflow (for Unix Makefiles) with exact commands that I see at the moment:

Clone project, checkout development branch:
- git clone git@github.com:Hintzelab/MABE.git
- cd MABE
- git checkout development
CMake the makefiles
- cmake ./ -G Unix\ Makefiles
Make your changes
Make the executable:
- make
Run, repeating 3 & 4 as needed

cliff-bohm commented 4 years ago

but I thought you were suggesting in addition to this that it would be better to create a build directory and run from there?

also where does changing what modules we are building with come into play? is cmake looking at build options?

JorySchossau commented 4 years ago

It does not maintain the old behavior so it really should have had a review period. reverted, so we can do that.

there is now some text at the bottom of build options, but it does not say anything along the lines of "If you are doing a normal build the old fashion way you can ignore this" that sounds like a good idea.

I think that the default behavior for python pythonTools/mbuild.py

should be to generate an executable. This is an approachability issue for new users as must as anything else. A user's first experience with MABE is, pull, run mbuild, run mabe. It's nice and simple.

I agree. I also don't want people to touch mbuild. cmake make is the workflow I'm proposing, which is bolstered by any other software / stackoverflow they'll ever see on the subject, except mabe is even easier because you don't need the install step. From a naive user's perspective, one extra command is definitely more, but the weeks of work I would need to put in to continue supporting the custom projects is well worth the change to avoid.

If we want to have the code live in one directory and the executable get generated somewhere else I'm fine with that. If we are changing the layout of directories, I would prefer that we set it up such that we have:
/code
/make and object (i.e. where the build happens) this could be in /code of course
/work (where the executable gets placed)

This is something that we get to specify in the wiki, because it doesn't matter where the build directory is. If I'm feeling lazy I might even put it in the mabe dir.

and we run mbuild.py from the directory above. This way if I don't care about code or the make process I don't need to look at it. I just mbuild and go into work to get my executable. I also like this layout because it supports copying the work directory to create experiments.

I'm all on board with that.

That all being said... Can you spell out in one message the exact commands I need to run to see the new workflow. I'm still not getting it.

Sure, git clone git@github.com:hintzelab/mabe mabe mkdir build cd build cmake ../mabe make -j10 or as simple as git clone git@github.com:hintzelab/mabe mabe cmake mabe make -j10

JorySchossau commented 4 years ago

Also, if buildOptions changes, then cmake needs to run again.

cliff-bohm commented 4 years ago

Okay, so just so I am totally clear, cmake fully replaces mbuild.py... including looking at buildOptions.txt?

(Also, sorry, I did not notice that there were two of you answering...)

JorySchossau commented 4 years ago

As far as the user is concerned, cmake replaces mbuild. We still require it to parse buildOptions and generate modules.h for the time being but they don't need to know that. It also reads in any user-specified cmake fragments from buildOptions and concatenates them for the last phase of the configuration. This allows integrating with other software with only a single line change in buildOptions.

cliff-bohm commented 4 years ago

Okay, I'm starting to get it now (also looking at your e-mail helped understand the needs for this change better).

Can we have a conversation about standardizing the directory layout today?

also... Looks like the command line prompt for vs is only 19. We will need to add language somewhere that effect.

I'm still on 17... so I'm going to update to 19 now. (this will take a minute....)

cliff-bohm commented 4 years ago

Help...

in a regular terminal:

$ cmake ../MABE -G
-bash: cmake: command not found

in Developer Command Prompt for VS 2019:

C:\Users\cliff\Desktop\TEST_NEW_PROCESS>cmake ../MABE -G
CMake Error: No generator specified for -G

Generators
* Visual Studio 16 2019        = Generates Visual Studio 2019 project files.
                                 Use -A option to specify architecture.
  Visual Studio 15 2017 [arch] = Generates Visual Studio 2017 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 14 2015 [arch] = Generates Visual Studio 2015 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 12 2013 [arch] = Generates Visual Studio 2013 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 11 2012 [arch] = Generates Visual Studio 2012 project files.
                                 Optional [arch] can be "Win64" or "ARM".
  Visual Studio 10 2010 [arch] = Generates Visual Studio 2010 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Visual Studio 9 2008 [arch]  = Generates Visual Studio 2008 project files.
                                 Optional [arch] can be "Win64" or "IA64".
  Borland Makefiles            = Generates Borland makefiles.
  NMake Makefiles              = Generates NMake makefiles.
  NMake Makefiles JOM          = Generates JOM makefiles.
  MSYS Makefiles               = Generates MSYS makefiles.
  MinGW Makefiles              = Generates a make file for use with
                                 mingw32-make.
  Unix Makefiles               = Generates standard UNIX makefiles.
  Green Hills MULTI            = Generates Green Hills MULTI files
                                 (experimental, work-in-progress).
  Ninja                        = Generates build.ninja files.
  Watcom WMake                 = Generates Watcom WMake makefiles.
  CodeBlocks - MinGW Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles = Generates CodeBlocks project files.
  CodeBlocks - NMake Makefiles JOM
                               = Generates CodeBlocks project files.
  CodeBlocks - Ninja           = Generates CodeBlocks project files.
  CodeBlocks - Unix Makefiles  = Generates CodeBlocks project files.
  CodeLite - MinGW Makefiles   = Generates CodeLite project files.
  CodeLite - NMake Makefiles   = Generates CodeLite project files.
  CodeLite - Ninja             = Generates CodeLite project files.
  CodeLite - Unix Makefiles    = Generates CodeLite project files.
  Sublime Text 2 - MinGW Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - NMake Makefiles
                               = Generates Sublime Text 2 project files.
  Sublime Text 2 - Ninja       = Generates Sublime Text 2 project files.
  Sublime Text 2 - Unix Makefiles
                               = Generates Sublime Text 2 project files.
  Kate - MinGW Makefiles       = Generates Kate project files.
  Kate - NMake Makefiles       = Generates Kate project files.
  Kate - Ninja                 = Generates Kate project files.
  Kate - Unix Makefiles        = Generates Kate project files.
  Eclipse CDT4 - NMake Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - MinGW Makefiles
                               = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Ninja         = Generates Eclipse CDT 4.0 project files.
  Eclipse CDT4 - Unix Makefiles= Generates Eclipse CDT 4.0 project files.

C:\Users\cliff\Desktop\TEST_NEW_PROCESS>

So, how do I specify the Generator?

JorySchossau commented 4 years ago

If you leave the -G off, then it does the '*' item by default. Otherwise you can specify a specific one: -G "Visual Studio 10 2010" and maybe you need the architecture on the end too, which might be x86_64.

cliff-bohm commented 4 years ago

I assume you mean -G "Visual Studio 16 2019" putting it in quotes was not obvious.

Now, I get:

CMake Error: The source directory "C:/Users/cliff/Desktop/TEST_NEW_PROCESS" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.

JorySchossau commented 4 years ago

I mean, you can leave off the -G "Visual Studio 16 2019" entirely, and it'll do the 'right thing' by default and make you a visual studio project. So you only need to type cmake ../mabe and it makes you a vcxproj file. However, this only works if you're using the Visual Studio Command Line environment found in the start menu. But that's what someone should be doing anyway if cmd.exe is their only shell. But we should be telling users to install the new command line by MS, as it provides a superior experience.

cliff-bohm commented 4 years ago

I'll try this again... it did not seem to be working, but maybe I was being dumb. One thing worth mentioning here, the commands in the command line environment (I assume you mean the developer command prompt - if not, I'm not using the correct thing) are the windows not unix/linux commands... which is... crapy since it makes the user experience change significantly based on the system you happen to be working on.

Hintzelab / MABE

added cmake out of src workflow #325