Since #3891 we've observed occasional failures in nightly CI runs that test the upgrade/downgrade functionality. #3913 included a fix I suggested that didn't actually help, so I spent more time thinking about the cause.
As part of a long-ago fix in #1291, the build system emits marker files for every RPM that comes out of a package build. Prior to the static-pods extraction, we would end up with marker files like this:
Those marker files act as an instruction to buildsys to remove the corresponding files from build/rpms whenever the package - in this case, os - is rebuilt.
Following the refactor, we now emit these files for static-pods:
The bug happens as a result of this sequence of events:
In a local workspace, check out a commit from before the package was factored out.
Run the build, which creates markers under os for the RPMs.
Check out a commit from after the package was factored out.
Run the build again.
If static-pods finishes building before os starts building, then when os starts, buildsys will remove the RPMs that the static-pods package generated, because os still has marker files for those RPMs.
The following script can reproduce the bug fairly reliably, by limiting the max parallel builds to one, which makes it more likely to hit the problematic sequence:
#!/bin/bash
cargo make clean
export BUILDSYS_VARIANT=aws-k8s-1.23
git checkout upstream/1.19.x
cargo make
git checkout develop
cargo make -e BUILDSYS_JOBS=1
The fix is to change the versions of any package that previously existed as a subpackage. That way, any marker files from an older build of the previous parent package will not find any output to remove.
Since #3891 we've observed occasional failures in nightly CI runs that test the upgrade/downgrade functionality. #3913 included a fix I suggested that didn't actually help, so I spent more time thinking about the cause.
As part of a long-ago fix in #1291, the build system emits marker files for every RPM that comes out of a package build. Prior to the static-pods extraction, we would end up with marker files like this:
Those marker files act as an instruction to
buildsys
to remove the corresponding files frombuild/rpms
whenever the package - in this case,os
- is rebuilt.Following the refactor, we now emit these files for
static-pods
:The bug happens as a result of this sequence of events:
os
for the RPMs.static-pods
finishes building beforeos
starts building, then whenos
starts,buildsys
will remove the RPMs that thestatic-pods
package generated, becauseos
still has marker files for those RPMs.The following script can reproduce the bug fairly reliably, by limiting the max parallel builds to one, which makes it more likely to hit the problematic sequence:
The fix is to change the versions of any package that previously existed as a subpackage. That way, any marker files from an older build of the previous parent package will not find any output to remove.