search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.22k stars 1.25k forks source link

replace PODS build system with cmake external projects #1169

Closed RussTedrake closed 9 years ago

RussTedrake commented 9 years ago

@tkoolen points out that many features require cmake >= 3.2 and ubuntu is on 2.8 still.

we could get from source: http://www.cmake.org/files/v3.2/cmake-3.2.3-Linux-x86_64.tar.gz or could add a ppa that has 3.2 prebuilt. http://newtips4u.com/how-to-install-cmake-3-2-on-ubuntu-14-04/

patmarion commented 9 years ago

which features are you using the require 3.2? I've found 2.8 to have all the features I need so far.

tkoolen commented 9 years ago

Copying from email discussion: first pass at this.

Pretty rough around the edges, but it's a start. Tested these instructions under ubuntu. Should also work on OSX, but somehow it didn't work from the command line on my machine due to some issues I've been having.

Cmake

First download cmake 3.2.3 from http://www.cmake.org/files/v3.2/cmake-3.2.3-Linux-x86_64.tar.gz unzip, and wherever it says cmake below, replace with unzipped cmake executable. Ubuntu's cmake is at 2.8 and several ExternalProject features were only introduced in 3.2.

Create directories

mkdir drake-cmake-test # where the code lives mkdir drake-cmake-test-build # where the build happens mkdir drake-cmake-test-install # where stuff gets installed to

Clone drake

cd drake-cmake-test git clone https://github.com/tkoolen/drake cd drake git submodule update --init --recursive git checkout cmake-project-fixes cd cmake git remote add tkoolen https://github.com/tkoolen/cmake git fetch tkoolen git checkout cmake-project-fixes cd ../../

Set up and build (dependencies first, for now)

download and unzip https://www.dropbox.com/s/auu1bf8pufil8ta/drake-cmake-files.zip?dl=0 copy CMakeLists.txt and libbotCMakeLists.txt into drake-cmake-test following entry is currently necessary because the drake project doesn't have the dependency on LCM set up correctly edit CMakeLists.txt, comment out last line cd ../drake-cmake-test-build cmake ../drake-cmake-test -DCMAKE_INSTALL_PREFIX=../drake-cmake-test-install make -j1 # j8 doesn't quite work yet because of prompts. Note: if you get stuck on bertini login, ctrl+c, git pull in bertini, then make again

Build drake

edit ../drake-cmake-test/CMakeLists.txt uncomment the last line again cd ../drake-cmake-test-build cmake ../drake-cmake-test -DCMAKE_INSTALL_PREFIX=../drake-cmake-test-install make -j1

tkoolen commented 9 years ago

I'll boot into linux in a little bit so I can check what exactly goes wrong with CMake 2.8.

psiorx commented 9 years ago

I'm doing it now.

psiorx commented 9 years ago

Confirmed that this still works with CMake 2.8 (Just changed the cmake_minimum_required version to 2.8)

tkoolen commented 9 years ago

@patmarion mentioned that it's better to have all of the projects be external projects (so we should replace the add_subdirectory(drake) at the end by another ExternalProject_Add call. That would also allow us to properly set up the dependencies that drake has on the other projects (using the ExternalProject DEPENDS argument), which resolves the hack with uncommenting and commenting and should also enable parallel drake-distro builds (build the projects in drake-distro in parallel where possible instead of iterating over them serially).

By the way, this is no longer coupled to using CLion; I have it working nicely now by creating a project that just points to the drake CMakeLists.txt. I don't think this worked in the first versions of CLion I tried, but it does now.

RussTedrake commented 9 years ago

been playing a bit with this. it seems like using external projects is definitely cleaner that the PODS tobuild.txt logic, but that it doesn't fundamentally get around the use of pkg-config . Is that right?

For instance, I'm updating our libbot pod to use this, so at the root libbot level I have a file that just adds all of the subfolders as external projects. I was hoping that I could then go into the subprojects and have add target dependencies , or target_link_libraries, which directly referenced the targets from the other subprojects. But that doesn't seem to work.

The only targets that cmake knows about are the current project's targets. I can add dependencies in the root libbot project between the subprojects. And I could hypothetically add dependencies between a specific target and an external project, but that would only work for targets defined at the same project level as the external project was defined (the subproject's cmake doesn't know about the external projects defined at the libbot level).

So the build should be much cleaner, but we'll still depend on something like pkg-config to share dependencies across subprojects. The alternative might be to, e.g., get rid of drake-distro and have drake add the external projects directly and reference them. But that seems somehow less clean.

@patmarion , @tkoolen -- does this seem correct to you?

patmarion commented 9 years ago

External project does not provide an alternative to pkg-config for finding external dependencies. An alternative to pkg-config is cmake's find_library/find_path mechanism, or, better, -config.cmake files which are cmake's answer to .pc files.

For libbot, the tobuild.txt toplevel file configures and builds each subdirectory as a separate project. The dependencies between them are handled with .pc files and pkg-config. If we are discussing changing libbot's build system to pure cmake, then I would propose that there is a single top level CMakeLists.txt and that all the subdirectories are part of the same project. It would be easy to do since libbot is already quite modularized. Then there is no requirement to use pkg-config, it's all one project. Libbot would install a libbot-config.cmake file and any cmake project could use libbot simply by calling cmake's find_package. Whether you chose to maintain a .pc file for the libbot libs would be your choice.

tkoolen commented 9 years ago

In fact, I made a CMakeLists.txt for libbot (libbotCMakeLists.txt above) that just calls add_subdirectory (not ExternalProject_Add) for all of libbot's subdirectories, i.e.:

project(libbot)

add_subdirectory(bot2-core)
add_subdirectory(bot2-vis)
add_subdirectory(bot2-procman)
add_subdirectory(bot2-lcmgl)
add_subdirectory(bot2-lcm-utils)
add_subdirectory(bot2-param)
add_subdirectory(bot2-frames)

I did this in part to play with the PATCH_COMMAND functionality of ExternalProject_Add: I made the top level CMakeLists.txt copy libbotCMakeLists.txt to the root of libbot after cloning it.

patmarion commented 9 years ago

Yep, like that. But, that doesn't actually work... because now you are configuring bot2-vis before bot2-core has been compiled. So how can bot2-vis find bot2-core? bot2-core.pc doesn't exist yet.

RussTedrake commented 9 years ago

got it. i guess we don't loose anything this way? if the subfolders call the project() command again themselves, is that going to screw things up? i'd hate to loose the current modularity of it. On Jul 17, 2015, at 10:33 AM, Twan Koolen notifications@github.com wrote:

In fact, I made a CMakeLists.txt for libbot (libbotCMakeLists.txt above) that just calls add_subdirectory (not ExternalProject_Add) for all of libbot's subdirectories, i.e.:

project(libbot)

add_subdirectory(bot2-core) add_subdirectory(bot2-vis) add_subdirectory(bot2-procman) add_subdirectory(bot2-lcmgl) add_subdirectory(bot2-lcm-utils) add_subdirectory(bot2-param) add_subdirectory(bot2-frames) I did this in part to play with the PATCH_COMMAND functionality of ExternalProject_Add: I made the top level CMakeLists.txt copy libbotCMakeLists.txt to the root of libbot after cloning it.

— Reply to this email directly or view it on GitHub.

RussTedrake commented 9 years ago

FWIW - this is my first pass at it: https://github.com/RobotLocomotion/libbot/pull/5

RussTedrake commented 9 years ago

I do think that the calls to project() in the subdirectories would cause problems. But if we don't call them (could have it be optional), then other things will break (e.g. the lcmtypes_projectname convention).
I know my solution above is only halfway to where it could be, but I think it might be a good local minima?

RussTedrake commented 9 years ago

I've played with this more now and can get a lot of things to work. But i'm now mostly unconvinced that it is better than PODS.

Advantages:

Disadvantages:

Things i thought it might help with, but doesn't:

So in the end :

So i'm on the fence. Any other thoughts?

RussTedrake commented 9 years ago

notes about resolving most of my qualms above:

cmake introduced the BUILD_ALWAYS flag in v 3.1, which would make the workflow more familiar (that's effectively what we have with the PODS make tobuild.txt). I could make a version that worked in 2.8 by appropriate messing with stamp files, but will not go there today.

i could manually set the download location and/or otherwise make sure that make clean doesn't zap the downloads. definitely should be possible.

the more recommended way to force a build in the subproject is not to zap the pod-build folder but to zap the stampfiles.

my mods to the repos will all be in git history, but I'm not going to push them into the main workflow. the pods system seems better to me for now. (for libbot, see a revert in the history of master; for lcm-pod, eigen-pod, and drake-distro, look for a branch called ExternalProject).

The only disadvantage I can see is that we don't get to use CLion?

tkoolen commented 9 years ago

I mentioned in an earlier comment:

By the way, this is no longer coupled to using CLion; I have it working nicely now by creating a project that just points to the drake CMakeLists.txt. I don't think this worked in the first versions of CLion I tried, but it does now.

patmarion commented 9 years ago

my 2 cents:

The the biggest disadvantage and primary reason not to switch away from PODS is the first one you cited: it changes the current workflow, introduces new concepts, and will require time & effort to work all the bugs out.

I think that it's not worth changing the build system if you are happy with the current one, and the current one supports the platforms you care about. There are currently 17 projects built before drake, and they have had years to be tuned to work correctly as PODS. It could take an equal amount of time to deploy an equivalent pure cmake version of that system.

However, I think some of the disadvantages you listed may be due to incorrect implementation, and differences of opinion on what you want from a build system. The ExternalProject functionality is just one part, there are other important changes required to make a pure cmake build system that is equivalent to PODS.

RussTedrake commented 9 years ago

Totally agree. I definitely see the limitations in pods. I just don't think externalproject addresses them. If there are any errors in my assessment, i'd definitely love to hear that now (that's the point of posting these :).

To be clear, i think externalproject in cmake > 3.1 is a lot closer. So part of the constraint is trying to support cmake 2.8.

On Jul 19, 2015, at 1:15 PM, Pat Marion notifications@github.com wrote:

my 2 cents:

The the biggest disadvantage and primary reason not to switch away from PODS is the first one you cited: it changes the current workflow, introduces new concepts, and will require time & effort to work all the bugs out.

I think that it's not worth changing the build system if you are happy with the current one, and the current one supports the platforms you care about. There are currently 17 projects built before drake, and they have had years to be tuned to work correctly as PODS. It could take an equal amount of time to deploy an equivalent pure cmake version of that system.

However, I think some of the disadvantages you listed may be due to incorrect implementation, and differences of opinion on what you want from a build system. The ExternalProject functionality is just one part, there are other important changes required to make a pure cmake build system that is equivalent to PODS.

— Reply to this email directly or view it on GitHub.

rdeits commented 9 years ago

Actually, it turns out that it is possible to force an external project to build always even with cmake 2.8. You just have to add something like this right after the ExternalProject_Add call:

ExternalProject_Add_Step(iris_project forceconfigure COMMAND ${CMAKE_COMMAND} -E echo "Force configure of iris" DEPENDEES update DEPENDERS configure ALWAYS 1)

The important part is that you replace iris_project with whatever the name of your external project is.

On Sun, Jul 19, 2015 at 3:32 PM, Russ Tedrake notifications@github.com wrote:

Totally agree. I definitely see the limitations in pods. I just don't think externalproject addresses them. If there are any errors in my assessment, i'd definitely love to hear that now (that's the point of posting these :).

To be clear, i think externalproject in cmake > 3.1 is a lot closer. So part of the constraint is trying to support cmake 2.8.

On Jul 19, 2015, at 1:15 PM, Pat Marion notifications@github.com wrote:

my 2 cents:

The the biggest disadvantage and primary reason not to switch away from PODS is the first one you cited: it changes the current workflow, introduces new concepts, and will require time & effort to work all the bugs out.

I think that it's not worth changing the build system if you are happy with the current one, and the current one supports the platforms you care about. There are currently 17 projects built before drake, and they have had years to be tuned to work correctly as PODS. It could take an equal amount of time to deploy an equivalent pure cmake version of that system.

However, I think some of the disadvantages you listed may be due to incorrect implementation, and differences of opinion on what you want from a build system. The ExternalProject functionality is just one part, there are other important changes required to make a pure cmake build system that is equivalent to PODS.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-122697169 .

Robin L. H. Deits Robot Locomotion Group @CSAIL Massachusetts Institute of Technology

RussTedrake commented 9 years ago

ah. that’s much better than manually touching the stamp files!

On Jul 20, 2015, at 2:00 PM, Robin Deits notifications@github.com wrote:

Actually, it turns out that it is possible to force an external project to build always even with cmake 2.8. You just have to add something like this right after the ExternalProject_Add call:

ExternalProject_Add_Step(iris_project forceconfigure COMMAND ${CMAKE_COMMAND} -E echo "Force configure of iris" DEPENDEES update DEPENDERS configure ALWAYS 1)

The important part is that you replace iris_project with whatever the name of your external project is.

On Sun, Jul 19, 2015 at 3:32 PM, Russ Tedrake notifications@github.com wrote:

Totally agree. I definitely see the limitations in pods. I just don't think externalproject addresses them. If there are any errors in my assessment, i'd definitely love to hear that now (that's the point of posting these :).

To be clear, i think externalproject in cmake > 3.1 is a lot closer. So part of the constraint is trying to support cmake 2.8.

On Jul 19, 2015, at 1:15 PM, Pat Marion notifications@github.com wrote:

my 2 cents:

The the biggest disadvantage and primary reason not to switch away from PODS is the first one you cited: it changes the current workflow, introduces new concepts, and will require time & effort to work all the bugs out.

I think that it's not worth changing the build system if you are happy with the current one, and the current one supports the platforms you care about. There are currently 17 projects built before drake, and they have had years to be tuned to work correctly as PODS. It could take an equal amount of time to deploy an equivalent pure cmake version of that system.

However, I think some of the disadvantages you listed may be due to incorrect implementation, and differences of opinion on what you want from a build system. The ExternalProject functionality is just one part, there are other important changes required to make a pure cmake build system that is equivalent to PODS.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-122697169 .

Robin L. H. Deits Robot Locomotion Group @CSAIL Massachusetts Institute of Technology — Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-122968669.

patmarion commented 9 years ago

I'm not quite understanding the issue. Why do you have to force a build? In the external project setups I've done, everything "just works", I've not had to do any funny business. It always runs through the required steps after modifications to a sub-project.

patmarion commented 9 years ago

Oh I guess you don't have any external projects with an update command. Usually the superbuilds I setup use external_project to checkout projects, so it always steps through the update/configure/make/install sequence. Yes, you may need the old force reconfigure trick that @rdeits posted. I'd recommend wrapping with macro since you may need to put it on all the projects.

RussTedrake commented 9 years ago

Short version:
I’ve got a version of drake-distro using cmake’s ExternalProject that I like. I’m hoping you can start using it so that we can debug the workflow. Just flip to the “ExternalProject” branch of drake-distro, uncomment your preferred lines at the top of CMakeLists.txt and give it a shot.

Longer version: Maintaining many different branches of drake-distro with different submodule refs is a pain in the ***. The new instructions for the wiki would be:

git clone https://github.com/RobotLocomotion/drake-distro.git

edit CMakeLists.txt and uncomment the packages you want near the top of the file.

make

That way people can select exactly which packages they want, and cmake will git clone them as appropriate. No more mysterious permission issues (that will be documented in the CMakeLists.txt). Now we only have to update the submodule refs in one place. And cmake’s overpowering “return to commit” logic will probably make things easier for helping novices fix their problems. (I’ve spent lots of time helping people do “git submodule update —init —recursive”, etc).

To maintain our current workflow, I just have cmake treat all of the subprojects as pods (not cmake projects). That was the easiest way for me to make sure we can build once in drake-distro then cd in and do all of our work in drake, building locally like we’re used to. I think I’ve caught most of the use cases, but am hoping you guys can test it out before I make the switch. Feedback appreciated.

BTW - Robin’s trick for forcing the configure worked well — thanks for that, it made the difference.

tkoolen commented 9 years ago

Just tried this out on osx with all projects uncommented except for gtk. A couple of comments:

patmarion commented 9 years ago

Worked for me! Really nice, I like the custom clean commands. I also like the optional_dependencies handling. (It does require the subproject behavior to be: search for dependency and use it if found.)

Have you thought about introducing cmake options to replicate the different distro branches? Instead of flipping individual dependencies, maybe define a few common selections?

What about using an empty string for CONFIGURE_COMMAND since the pod makefile does it automatically:

CONFIGURE_COMMAND ""

I'd also suggest that you pass cmake variables as environment variables to the POD makefiles:

BUILD_COMMAND $(MAKE) BUILD_PREFIX=${CMAKE_INSTALL_PREFIX} BUILD_TYPE=${CMAKE_BUILD_TYPE}

Note, I used $(MAKE) instead of make. This allows parallel builds to work correctly (-j4). Not sure if $(MAKE) also works on Windows?

I got the director to build, but for some reason this didn't work:

set_target_properties(director PROPERTIES SOURCE_DIR ${PROJECT_SOURCE_DIR}/director/distro/pods/drake-distro)

I had to do something a bit funny, I did this as a workaround:

set(director_sub_source_dir /distro/pods/drake-distro)
...
SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}${${proj}_sub_source_dir}

After adopting ExternalProject, you might consider supporting non-POD projects. This is how some of the externals in DRC are handled now. And you can remove from POD wrappers. For example, lcm, eigen, director, bullet, can be built directly via cmake instead of POD makefiles.

patmarion commented 9 years ago

@tkoolen try:

make clean-all

or for an individual project:

make clean-<proj>

CMake doesn't let you override the default clean command to do custom behavior. But @RussTedrake might be able to override the drake-distro pod Makefile to forward a "make clean" to pod-build make clean-all.

patmarion commented 9 years ago

Oops, looks like Russ already did put support for the clean command in the drake-distro Makefile. Perhaps the error was in one of the subprojects you uncommented?

patmarion commented 9 years ago

If I comment out the forceconfigure custom step, it seems to still have the correct behavior, perhaps I am missing something. What should I look for to recreate the issue?

tkoolen commented 9 years ago

Yeah, I got rid of that bullet point, but I wasn't quick enough. The error during clean was just because the build before that failed.

tkoolen commented 9 years ago

Pat,

I had to do something a bit funny, I did this as a workaround:

set(director_sub_source_dir /distro/pods/drake-distro)
...
SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}${${proj}_sub_source_dir}

I don't quite understand what I need to change in the CMakeLists.txt to get director to build. Could you elaborate or post your modified CMakeLists.txt somewhere?

tkoolen commented 9 years ago 
CONFIGURE_COMMAND ""

This is what actually allowed me to build director.

patmarion commented 9 years ago

My changes are here. Perhaps the source dir property issue is only for cmake 2.8 (im testing on ubuntu 14.04)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 7a3cb7e..96bd77b 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -79,6 +79,8 @@ set(yalmip_GIT_TAG c071fb7b7193491f8eefadba3bfff26160ad6cd4)
 set(xfoil_GIT_REPOSITORY https://github.com/RobotLocomotion/xfoil.git)
 set(xfoil_GIT_TAG fde0a9368dd451c93604051fc5704e120073800c)

+set(director_sub_source_dir /distro/pods/drake-distro)
+
 foreach (proj ${EXTERNAL_PROJECTS})
        set(deps ${${proj}_dependencies})
        foreach(dep ${${proj}_optional_dependencies})
@@ -90,13 +92,13 @@ foreach (proj ${EXTERNAL_PROJECTS})

        message(STATUS "Preparing to build ${proj} with dependencies: ${deps}")
        ExternalProject_Add(${proj} 
-               SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}
+               SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}${${proj}_sub_source_dir}
                GIT_REPOSITORY ${${proj}_GIT_REPOSITORY}
                GIT_TAG ${${proj}_GIT_TAG}
-               CONFIGURE_COMMAND make configure
+               CONFIGURE_COMMAND ""
                BUILD_IN_SOURCE 1
 #              BUILD_ALWAYS 1
-               BUILD_COMMAND make
+               BUILD_COMMAND $(MAKE) BUILD_PREFIX=${CMAKE_INSTALL_PREFIX} BUILD_TYPE=${CMAKE_BUILD_TYPE}
                INSTALL_COMMAND ""
                DEPENDS ${deps}
                )
tkoolen commented 9 years ago

Thanks. About using BUILD_PREFIX=${CMAKE_INSTALL_PREFIX}, this variable defaults to /usr/local on UNIX and c:/Program Files on Windows. Do we want that?

patmarion commented 9 years ago

The drake-distro Makefile will create a build directory and pass it in to cmake as CMAKE_INSTALL_PREFIX automatically. The code I added forwards this information into the POD sub projects. It works correctly already because POD makefiles will search for a build directory, but usually the tobuild.txt feature passes in the BUILD_PREFIX environment variable explicitly.

tkoolen commented 9 years ago

Ah OK.

On Fri, Jul 31, 2015 at 11:36 AM, Pat Marion notifications@github.com wrote:

The drake-distro Makefile will create a build directory and pass it in to cmake as CMAKE_INSTALL_PREFIX automatically. The code I added forwards this information into the POD sub projects. It works correctly already because POD makefiles will search for a build directory, but usually the tobuild.txt feature passes in the BUILD_PREFIX environment variable explicitly.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126726630 .

RussTedrake commented 9 years ago

I thought I had that working. The makefile defines a target clean that gets passed to cmake as clean-all.

On Jul 31, 2015, at 10:59 AM, Pat Marion notifications@github.com wrote:

@tkoolen https://github.com/tkoolen try:

make clean-all or for an individual project:

make clean- CMake doesn't let you override the default clean command to do custom behavior. But @RussTedrake https://github.com/RussTedrake might be able to override the drake-distro pod Makefile to forward a "make clean" to pod-build make clean-all.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126718930.

RussTedrake commented 9 years ago

those look like good changes to me. you can see that I set the source directory for director near the bottom, so we can zap that logic if we use your version.

On Jul 31, 2015, at 11:19 AM, Pat Marion notifications@github.com wrote:

My changes are here. Perhaps the source dir property issue is only for cmake 2.8 (im testing on ubuntu 14.04)

diff --git a/CMakeLists.txt b/CMakeLists.txt index 7a3cb7e..96bd77b 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -79,6 +79,8 @@ set(yalmip_GIT_TAG c071fb7b7193491f8eefadba3bfff26160ad6cd4) set(xfoil_GIT_REPOSITORY https://github.com/RobotLocomotion/xfoil.git) set(xfoil_GIT_TAG fde0a9368dd451c93604051fc5704e120073800c)

+set(director_sub_source_dir /distro/pods/drake-distro) + foreach (proj ${EXTERNAL_PROJECTS}) set(deps ${${proj}_dependencies}) foreach(dep ${${proj}_optional_dependencies}) @@ -90,13 +92,13 @@ foreach (proj ${EXTERNAL_PROJECTS})

    message(STATUS "Preparing to build ${proj} with dependencies: ${deps}")
    ExternalProject_Add(${proj}
  • SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}
  • SOURCE_DIR ${PROJECT_SOURCE_DIR}/${proj}${${proj}_sub_source_dir} GIT_REPOSITORY ${${proj}_GIT_REPOSITORY} GIT_TAG ${${proj}_GIT_TAG}
  • CONFIGURE_COMMAND make configure
  • CONFIGURE_COMMAND "" BUILD_IN_SOURCE 1

    BUILD_ALWAYS 1

  • BUILD_COMMAND make
  • BUILD_COMMAND $(MAKE) BUILD_PREFIX=${CMAKE_INSTALL_PREFIX} BUILD_TYPE=${CMAKE_BUILD_TYPE} INSTALL_COMMAND "" DEPENDS ${deps} )

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126723148.

patmarion commented 9 years ago

One potential workflow issue I just encountered: install_prereqs.sh requires the sub projects to exist, but those don't clone until build time now.

RussTedrake commented 9 years ago

yes — correct. i’ve got a note at the bottom of the cmakelists.txt to handle that one.

the other one I haven’t handled (yet) is the packaging file lists.

On Jul 31, 2015, at 12:00 PM, Pat Marion notifications@github.com wrote:

One potential workflow issue I just encountered: install_prereqs.sh requires the sub projects to exist, but those don't clone until build time now.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126734724.

patmarion commented 9 years ago

I would vote for using ssh urls by default (git@github.com) instead of https for the private repos, and add a cmake option to flip between ssh and https. But I understand reasons why you might prefer https, I don't care that much because I've used an alias to override it anyway.

rdeits commented 9 years ago

Just tried this out, and it looks good. I'd also vote for ssh urls (or perhaps we can make that a configuration option in the cmakelists), since otherwise I have to babysit the build process waiting for it to ask for my github password a dozen times.

-Robin

On Fri, Jul 31, 2015 at 12:14 PM, Pat Marion notifications@github.com wrote:

I would vote for using ssh urls by default (git@github.com) instead of https for the private repos, and add a cmake option to flip between ssh and https. But I understand reasons why you might prefer https, I don't care that much because I've used an alias to override it anyway.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126737486 .

Robin L. H. Deits Robot Locomotion Group @CSAIL Massachusetts Institute of Technology

rdeits commented 9 years ago

Also, the cmake repo inside drake still needs to be updated to get this fix: https://github.com/RobotLocomotion/cmake/commit/642bb825d7b806540b8345c26003a75be8c4a160

On Fri, Jul 31, 2015 at 1:36 PM, Robin Deits robin.deits@gmail.com wrote:

Just tried this out, and it looks good. I'd also vote for ssh urls (or perhaps we can make that a configuration option in the cmakelists), since otherwise I have to babysit the build process waiting for it to ask for my github password a dozen times.

-Robin

On Fri, Jul 31, 2015 at 12:14 PM, Pat Marion notifications@github.com wrote:

I would vote for using ssh urls by default (git@github.com) instead of https for the private repos, and add a cmake option to flip between ssh and https. But I understand reasons why you might prefer https, I don't care that much because I've used an alias to override it anyway.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126737486 .

Robin L. H. Deits Robot Locomotion Group @CSAIL Massachusetts Institute of Technology

Robin L. H. Deits Robot Locomotion Group @CSAIL Massachusetts Institute of Technology

rdeits commented 9 years ago

Ah, I see, cmake's up to date in the latest drake, but this distro just points to an older revision.

RussTedrake commented 9 years ago

yup

On Jul 31, 2015, at 1:40 PM, Robin Deits notifications@github.com wrote:

Ah, I see, cmake's up to date in the latest drake, but this distro just points to an older revision.

— Reply to this email directly or view it on GitHub https://github.com/RobotLocomotion/drake/issues/1169#issuecomment-126766654.

rdeits commented 9 years ago

So, what's the intended workflow for development on drake? If I do some work in drake and make a few commits, then run make in drake-distro, all of my commits will be discarded when the external project checks out its pegged revision of drake. Git does print a helpful warning when that happens:

[ 53%] Performing update step for 'drake'
Warning: you are leaving 1 commit behind, not connected to
any of your branches:

  df50cd6 add foo

If you want to keep them by creating a new branch, this may be a good time
to do so with:

 git branch new_branch_name df50cd6

HEAD is now at 6ed71b3... Merge pull request #1186 from RussTedrake/doc_to_underactuated

but it gets lost in the hundreds of other lines of output from the build process. For most people, that work will just be gone forever. That seems really bad, since it will be very difficult for anyone not super familiar with git to understand what happened or why all their work is gone.

Of course, since drake is still a pod, I can just run make inside the drake folder, but that won't work if we have any other pods that depend on drake (or if I want to work on a pod that drake depends on).

@patmarion is there some way to stop this from happening? This would be completely solved by using git submodules (which we can use with external projects), but I understand that maintaining the submodule references for all the drake versions is a pain.

RussTedrake commented 9 years ago

I imagined only running make inside the drake folder during development, but agree that it's the biggest downside. Could be a show-stopper? That's what I want to figure out.

I could easily make an "dev mode" option that you could pass to cmake that would disable the update step. but that's still a pain.

I just pushed more cmake logic to handle the director different clone / source dirs and to support install_prereqs.sh

rdeits commented 9 years ago

A similar problem is referenced here: http://public.kitware.com/pipermail/cmake-developers/2013-September/019873.html And I do think this is potentially a pretty big problem.

A "dev mode" would be possible, but disabling the update step would prevent the user from receiving (or even being aware of) changes to other pods that they're not currently working on.

Another option would be to use submodules but not have separate branches with different submodule configurations. For example, we could:

rdeits commented 9 years ago

Another alternative:

RussTedrake commented 9 years ago

I feel like I've spent lots of time helping people call git submodule update --init, etc, and was hoping that letting cmake force the git calls would make that go away.
The other (real) reason I started this today was that I'm trying to clean up the build server logic, because drake001 has been fried for a week now.