Open irfansharif opened 2 years ago
I’m imagining nothing more that invoking ps
early to see how many other dev
invocations there are and where the current process is when sorted by PID, using that number as part of an output base. There's another idea using tty
in this internal thread.
There are some problems with the approach in #85633, it had to be reverted (#85924, reverting #85858). Quoting @rickystewart from an internal thread:
I think at least three things need to change before this can be enabled by default:
- There must be a real, persistent way to opt out of the parallel output base behavior. An environment variable is fine for this purpose.
- The design is fine assuming that all you're doing is running builds and tests, but of course there are more bazel commands than that. In particular clean and shutdown have no clear meaning when this output base selection is happening transparently. If somebody runs bazel clean they should be sure that something has actually happened to the Bazel instance they expect.
- dev uses bazel info for various things assuming that the info it gets will be stable between subsequent bazel invocations, but with this change that is actually not true. For example, dev build calls bazel info bazel-bin to find out where the cockroach binary was placed, but in the presence of this change there is actually no guaranteeing that the subsequent bazel build it performs will use the same output base so the binary might end up in a different place. Then it will try to copy from a cockroach binary that doesn't actually exist in the best-case scenario, and in the worst-case scenario it will copy from an old cockroach binary that isn't up-to-date with your current code which will be very confusing.
We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!
Is your feature request related to a problem? Please describe.
Bazel doesn't let you run concurrent invocations: https://github.com/bazelbuild/bazel/issues/532. Two problems for us:
make {build,test}
, etcgopls
powered editors, you'd perhaps want to make use ofGOPACKAGESDRIVER
(see https://github.com/bazelbuild/rules_go/wiki/Editor-and-tool-integration) to have your IDE/editor learn to use bazel generated artifacts instead of redoing it throughgo build
. If limited to one bazel invocation, you're unable to use bazel while your IDE is off indexing/querying bazel for thingsDescribe the solution you'd like
Specifying a different
--output_base
lets you make use of different bazel servers: https://bazel.build/docs/user-manual#output-base. There's a bit of set up latency the first time a fresh output base is used, but it's possible to do the following two things in parallel:Perhaps we could maintain a bounded (up to 4?) free list of output bases to use, maybe similar to the directories we maintain for
dev cache
. We want to be re-using the output bases since there's non-negligible set up time when using one for the first time (maybe somethingdev {doctor,setup}
just does to get the four servers up and running?). After the bazel servers are running though, we should be able to just use the shared bazel-remote cache. For IDEs/editors specifically, perhaps we want instructions on how to have them use an output base other than the ones used bydev
.Jira issue: CRDB-15510
Epic CRDB-17171