search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.93k stars 1.06k forks source link

Jenkins build sometimes fails in postgres #5380

Open svarnau opened 4 years ago

svarnau commented 4 years ago

Standard error from external program {{ make MAKELEVEL=0 -j 80 }} running in '/nfusr/centos-gcp-cloud/jenkins-worker-4a1/jenkins/jenkins-github-yugabyte-db-centos-master-clang-release-1176/build/release-clang-dynamic-ninja/postgres_build', saving stdout to {{ /nfusr/centos-gcp-cloud/jenkins-worker-4a1/jenkins/jenkins-github-yugabyte-db-centos-master-clang-release-1176/build/release-clang-dynamic-ninja/postgres_build/make.out }}, stderr to {{ /nfusr/centos-gcp-cloud/jenkins-worker-4a1/jenkins/jenkins-github-yugabyte-db-centos-master-clang-release-1176/build/release-clang-dynamic-ninja/postgres_build/make.err }}: cat: access/objfiles.txt: No such file or directory cat: bootstrap/objfiles.txt: No such file or directory cat: catalog/objfiles.txt: No such file or directory cat: parser/objfiles.txt: No such file or directory cat: commands/objfiles.txt: No such file or directory cat: executor/objfiles.txt: No such file or directory cat: foreign/objfiles.txt: No such file or directory cat: lib/objfiles.txt: No such file or directory ...

svarnau commented 4 years ago

I looked through a number of build logs looking for patterns. Found several builds that had real compile errors. Found one that seemed to be a network issue (no route to host). I also found a few that seemed to be incorrect dependency files, such as:

.deps/fmgrtab.Po:139: *** missing separator. Stop.

These errors seem to most likely be parallelism or network issues.

svarnau commented 4 years ago

The generation of dependency files could be causing this error with parallel make, if a .Po dependency file is being written while another make process is trying to read it, we could see makefile format error.

The postgres makefile has an option to not generate dependencies, which might be good for jenkins running a clean build, but it is not clear if that changes other compiler configuration.

svarnau commented 4 years ago

Trial build with autoupdate=no looks very promising to build postgres without dependency files. Would need to make that optional flag that jenkins can use.

svarnau commented 4 years ago

Changing build to not enable dependencies works okay with release/debug builds but not with asan/tsan builds. Alternative approach is to leave dependencies alone and retry postgres build if it hits this error.

mbautin commented 3 years ago

I think we should not close this because the current workaround does not address the root cause.