Open knz opened 3 years ago
Another reason why it would be desirable to have a build target as prereq for the other targets: in case the source doesn't build, we would only hog 1 agent to report that error and abort the entire CI pipeline, instead of spawning 8 agents upfront.
Here's a sketch of a dep tree:
builder.sh mkrelease linux-gnu
(instead of make install
), then stores the produced binary somewhere where the other targets can find it. Possibly also runs make install
separately to ensure the binary produced by mkrelease
can be installed.Then we can remove the "Build test binary" step altogether.
Since the CI config change and the script updates are not atomic, we'll need to have a "transition phase" where the scripts work in both configurations. For this we can modify the scripts to expect a pre-built executable in a fixed path, and use if ! test -e path/to/executable; then ...build...; fi
to build it on demand if it's not there yet.
We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!
I just noticed that every CI runs compiles the entire crdb binary, including all web assets etc, a minimum of 7 times:
The acceptance CI target runs
teamcity-acceptance.sh
which in turn runspkg/acceptance/prepare.sh
which in turn runsbuilder.sh mkrelease linux-gnu
In other words, the acceptance CI target builds its own crdb executable.
Meanwhile, the Build test binary CI target runs
teamcity-build-test-binary.sh
which in turn runsbuilder.sh mkrelease linux-gnu
In other words, the "build test binary" CI target builds its own linux crdb executable.
Meanwhile, the roachtest CI target runs
teamcity-local-roachtest.sh
which in turn runsbuilder.sh make build
In other words, the roachtest CI target builds its own linux crdb executable.
Meanwhile, the Compile builds CI target runs
teamcity-compile-builds.sh
which in turn runspkg/cmd/compile-builds
which in turn runsrelease.MakeReleases
with "all targets" as arguments, which in turns runbuilder.sh mkrelease
on each supported target.In other words, the "Compile builds" CI target builds 3+ different crdb executables, including the
linux-gnu
one.Meanwhile, the verify archive CI target runs
teamcity-verify-archive.sh
which in turns create a tar archive of the source directory, then extracts it, then checks thatmake clean && make install
works on it.In other words, the "verify archive" CI target builds its own linux crdb executable.
We could shave some time on the entire CI run (as well as disk space, time collecting artifacts etc) by only building the linux executable just one time as pre-requisite for other CI targets.
I would recommend using verify archive as a starting point, since we also want to assert that the code can be built from a downloadable source archive.
cc @jlinder for triage
Epic CRDB-10439
Jira issue: CRDB-6973