dev,bazel: support concurrent invocations

[irfansharif]

Is your feature request related to a problem? Please describe.

Bazel doesn't let you run concurrent invocations: https://github.com/bazelbuild/bazel/issues/532. Two problems for us:

• regression from our make workflow where it was possible to run concurrent make {build,test}, etc
• for gopls powered editors, you'd perhaps want to make use of GOPACKAGESDRIVER (see https://github.com/bazelbuild/rules_go/wiki/Editor-and-tool-integration) to have your IDE/editor learn to use bazel generated artifacts instead of redoing it through go build. If limited to one bazel invocation, you're unable to use bazel while your IDE is off indexing/querying bazel for things

Another command (pid=34290) is running. Waiting for it to complete on the server (server_pid=13022)...

Describe the solution you'd like

Specifying a different --output_base lets you make use of different bazel servers: https://bazel.build/docs/user-manual#output-base. There's a bit of set up latency the first time a fresh output base is used, but it's possible to do the following two things in parallel:

# from dev test pkg/spanconfig/... --ignore-cache + specifying --output_base
$ bazel --output_base=/tmp/1 test pkg/spanconfig/...:all --nocache_test_results --test_env=GOTRACEBACK=all --test_output errors 

# from dev test pkg/ccl/spanconfigccl/... --ignore-cache + specifying --output_base
$ bazel --output_base=/tmp/2 test pkg/ccl/spanconfigccl/...:all --nocache_test_results --test_env=GOTRACEBACK=all --test_output errors

Perhaps we could maintain a bounded (up to 4?) free list of output bases to use, maybe similar to the directories we maintain for dev cache. We want to be re-using the output bases since there's non-negligible set up time when using one for the first time (maybe something dev {doctor,setup} just does to get the four servers up and running?). After the bazel servers are running though, we should be able to just use the shared bazel-remote cache. For IDEs/editors specifically, perhaps we want instructions on how to have them use an output base other than the ones used by dev.

Jira issue: CRDB-15510

Epic CRDB-17171

[irfansharif]

I’m imagining nothing more that invoking ps early to see how many other dev invocations there are and where the current process is when sorted by PID, using that number as part of an output base. There's another idea using tty in this internal thread.

[irfansharif]

There are some problems with the approach in #85633, it had to be reverted (#85924, reverting #85858). Quoting @rickystewart from an internal thread:

I think at least three things need to change before this can be enabled by default:

1. There must be a real, persistent way to opt out of the parallel output base behavior. An environment variable is fine for this purpose.
2. The design is fine assuming that all you're doing is running builds and tests, but of course there are more bazel commands than that. In particular clean and shutdown have no clear meaning when this output base selection is happening transparently. If somebody runs bazel clean they should be sure that something has actually happened to the Bazel instance they expect.
3. dev uses bazel info for various things assuming that the info it gets will be stable between subsequent bazel invocations, but with this change that is actually not true. For example, dev build calls bazel info bazel-bin to find out where the cockroach binary was placed, but in the presence of this change there is actually no guaranteeing that the subsequent bazel build it performs will use the same output base so the binary might end up in a different place. Then it will try to copy from a cockroach binary that doesn't actually exist in the best-case scenario, and in the worst-case scenario it will copy from an old cockroach binary that isn't up-to-date with your current code which will be very confusing.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!