bazel out-of-tree build fights with in-tree build

[RussTedrake]

I've got a workflow where i'm using the python bindings from a separate project. To do that, I've been using the out-of-source build instructions here: http://drake.mit.edu/python_bindings.html But I have to add some features in drake, too. So I'm often working in drake (via bazel), then building the library, and trying it from the other project.

Currently, calling bazel inside drake means that the next time I call make in drake-build, I have to rebuild everything (starting? with bullet, then multibody, etc). Same thing happens in the other direction. On this laptop, that costs me 10s of minutes.

Presumably, some of the bazel flags end up different on these two builds.

Based on a quick conversation w/ @EricCousineau-TRI , he pointed out that "Ah, I think that's because CMake actually uses a different compiler setup: https://github.com/RobotLocomotion/drake/blob/master/CMakeLists.txt#L53 @jamiesnape and @jwnimmer-tri have talked about making the default Drake-Bazel compiler reflect what CMake uses (based on CC, CXX, etc.) Er... But Bazel should be storing those two configs separately???"

Separate configs would be aok with me. But this one is a major productivity killer at the moment.

[jwnimmer-tri]

The mitigation, without waiting for a PR, would be to pass the same arguments to bazel when running it manually that CMake does when running it automatically. Line 108 is actually the place to look to find what that is -- I don't know if CMake is echo'ing that to you, but if you can find the value of BAZEL_COMMAND then that's your starting point.

[EricCousineau-TRI]

I think this would be another use-case for seeing if there's a way to tell Bazel "I don't care, just use whatever flags you had previously", rather than forcing you to remember everything via the command-line.

Perhaps another problem that could be solved with a tools/bazel wrapper?

[jwnimmer-tri]

Stickyness (hidden state) is an abomination. If we want an easy snippet of what the flags were, then we should have CMake emit an rcfile or wrapper script into its build tree that records what it did. Then if you want to do "Use CMake's cached flags, but with target name other than //:install", you have a mechanism to do that, which is explicit at the point of invocation.

[jamiesnape]

From memory, you need environment:

CC=${CMAKE_CURRENT_BINARY_DIR}/C_wrapper.sh CXX=${CMAKE_CURRENT_BINARY_DIR}/CXX_wrapper.sh

Options:

--compilation_mode=opt
--config=unsupported_crosstool
--cxxopt=-std=c++14
--host_cxxopt=-std=c++14

[jamiesnape]

We can do some of the with an rc file, though the environment variables may need a wrapper depending on how Bazel processes environment variables passed in an rc file.

[jwnimmer-tri]

Possibly also related, since macOS really only has one compiler anyway, I could see my way to making something analogous to unsupported_crosstool's behavior the macOS default -- that it uses whatever /usr/bin/ has installed, so that the CMake build doesn't have to override as much, and thus is easier to bounce back and forth.

[jamiesnape]

Actually the macOS default uses the results of xcrun, but I agree with the sentiment.

[jamiesnape]

Looks like you can set the compiler with --action_env=CC=/path/to/compiler so no wrapper needed. I will PR something shortly that writes an rc file. I just need to make sure Python still works on Mac when using it.

[jamiesnape]

Actually not so sure. I am seeing cc_wrapper.sh instead of CXX_wrapper.sh on some calls.

[EricCousineau-TRI]

Stickyness (hidden state) is an abomination

I'd honestly prefer some sort of stickiness, but I agree that hidden stickiness sucks. I would prefer that it be obvious / explicit, some flag like --use_previous_config.

I want this not just for CMake; for wrapper scripts, it's a minor annoyance to have to remember to build //tools/lint:buildifer (or something like external_data/cli) and then invoke the resultant binary. I have more explanation on this use case, but will table it for another issue / discussion.

[jwnimmer-tri]

Yup, I can get on board with "Freshen bazel-bin/** with what I need!" as meaningful a pain point.

[RussTedrake]

N.B. -- this is still a pain point (on ubuntu, too)

[jamiesnape]

More updates to ameliorate are upcoming (Monday or Tuesday). The idea is going to be to force a separate Bazel server install to run.

[jwnimmer-tri]

@RussTedrake I wonder if this pain point is resolved now?

[RussTedrake]

i have not tested this precise workflow, because my workflow has changed to installing from in-tree (using the bazel install target). My current situation is good, but I think not officially supported.

[jwnimmer-tri]

With recent changes, the two builds will not actively fight each other. They may not share cache hits, but at least they won't un-build each others' work. Good enough.