search

ros-infrastructure / buildfarm

Build scripts and notes for catkin debian build pipeline.
6 stars 15 forks source link

occationally builds fail due to ccache permission errors #174

Closed tfoote closed 9 years ago

tfoote commented 10 years ago

I've seen this once or twice in the past. A rebuild on the same machine will pass.

http://jenkins.ros.org/job/ros-groovy-depth-image-proc_binarydeb_quantal_amd64/190/console

/usr/lib/ccache/c++   -Ddepth_image_proc_EXPORTS -DROS_PACKAGE_NAME=\"depth_image_proc\" -DDISABLE_OPENNI -DDISABLE_OPENNI -Wno-deprecated -fPIC -isystem /usr/include/vtk-5.8 -isystem /opt/ros/groovy/include -isystem /opt/ros/groovy/include/opencv -isystem /opt/ros/groovy/include/pcl-1.6 -isystem /usr/include/eigen3 -isystem /usr/include/qhull    -o CMakeFiles/depth_image_proc.dir/src/nodelets/convert_metric.cpp.o -c /tmp/buildd/ros-groovy-depth-image-proc-1.10.13-0quantal-20140126-1902/src/nodelets/convert_metric.cpp
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/3/7: Permission denied
make[4]: *** [CMakeFiles/depth_image_proc.dir/src/nodelets/convert_metric.cpp.o] Error 1
make[4]: Leaving directory `/tmp/buildd/ros-groovy-depth-image-proc-1.10.13-0quantal-20140126-1902/obj-x86_64-linux-gnu'
make[3]: *** [CMakeFiles/depth_image_proc.dir/all] Error 2
make[3]: Leaving directory `/tmp/buildd/ros-groovy-depth-image-proc-1.10.13-0quantal-20140126-1902/obj-x86_64-linux-gnu'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/tmp/buildd/ros-groovy-depth-image-proc-1.10.13-0quantal-20140126-1902/obj-x86_64-linux-gnu'
dh_auto_build: make -j1 returned exit code 2
make[1]: *** [override_dh_auto_build] Error 2
make[1]: Leaving directory `/tmp/buildd/ros-groovy-depth-image-proc-1.10.13-0quantal-20140126-1902'
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
E: Failed autobuilding of package
tfoote commented 10 years ago

@vrabaud I filed this here.

tfoote commented 10 years ago

In this round we had several jobs fail the same way within a few minutes across several executors. They were all Quantal i386

https://groups.google.com/forum/#!topic/ros-buildfarm-release/0-9_KU55-GM

Started 1 hr 5 min ago
Took 1 min 55 sec on 64.91.234.21
Failed Build #12 (Jan 30, 2014 9:36:37 AM)

[ 81%] Building C object src/CMakeFiles/swiftnav-static.dir/coord_system.c.o 
cd /tmp/buildd/ros-hydro-swiftnav-0.0.4-0quantal-20140130-1738/obj-i686-linux-gnu/src && /usr/lib/ccache/gcc   -Wall -Wextra -Werror -std=gnu99  -I/tmp/buildd/ros-hydro-swiftnav-0.0.4-0quantal-20140130-1738/include    -o CMakeFiles/swiftnav-static.dir/coord_system.c.o   -c /tmp/buildd/ros-hydro-swiftnav-0.0.4-0quantal-20140130-1738/src/coord_system.c 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/6/8: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/8uEm-OkSeR0

Started 1 hr 4 min ago
Took 58 sec on 64.91.231.41
Failed Build #20 (Jan 30, 2014 9:38:07 AM)

  Building C object 
  CMakeFiles/cmTryCompileExec258286473.dir/testCCompiler.c.o 

  /usr/lib/ccache/gcc -o 
  CMakeFiles/cmTryCompileExec258286473.dir/testCCompiler.c.o -c 
  /tmp/buildd/ros-hydro-genpy-0.4.15-0quantal-20140130-1738/obj-i686-linux-gnu/CMakeFiles/CMakeTmp/testCCompiler.c 

  ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/b/7: Permission 
  denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/-K99nwVbaLw

Started 1 hr 7 min ago
Took 3 min 39 sec on 64.91.255.61
Failed Build #23 (Jan 30, 2014 9:35:31 AM)

[  1%] Building CXX object 3rdparty/openexr/CMakeFiles/IlmImf.dir/Imath/ImathFun.cpp.o 
cd /tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/obj-i686-linux-gnu/3rdparty/openexr && /usr/lib/ccache/c++   -DHAVE_CVCONFIG_H -fsigned-char -W -Wall -Werror=return-type -Werror=address -Werror=sequence-point -Werror=format-security   -Winit-self -Wpointer-arith  -Wsign-promo -Wno-narrowing -fdiagnostics-show-option -pthread -fomit-frame-pointer -msse2 -msse3 -mfpmath=sse -ffunction-sections -Wno-shadow -Wno-unused -Wno-sign-compare -Wno-undef -Wno-missing-declarations -Wno-uninitialized -Wno-switch -Wno-parentheses -Wno-array-bounds -Wno-extra -fPIC -O3 -DNDEBUG  -DNDEBUG -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/IlmImf -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/Imath -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/IlmThread -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/Iex -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/Half -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/obj-i686-linux-gnu/3rdparty/openexr -I/tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/obj-i686-linux-gnu    -o CMakeFiles/IlmImf.dir/Imath/ImathFun.cpp.o -c /tmp/buildd/ros-hydro-opencv2-doc-2.4.6-0quantal-20140130-1738/3rdparty/openexr/Imath/ImathFun.cpp 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/6/2: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/krS5xxEHyjo

Started 1 hr 6 min ago
Took 2 min 49 sec on 67.227.253.208
Failed Build #15 (Jan 30, 2014 9:36:49 AM)

[  5%] Building CXX object src/osgOcean/CMakeFiles/osgOcean.dir/Cylinder.cpp.o 
cd src/osgOcean && /usr/lib/ccache/c++   -DosgOcean_EXPORTS -DUSE_FFTW3F -DOSGOCEAN_LIBRARY -fPIC -I../../../include    -o CMakeFiles/osgOcean.dir/Cylinder.cpp.o -c ../../../src/osgOcean/Cylinder.cpp 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/d/c: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/Kmr-JAMlINg

Started 1 hr 4 min ago
Took 2 min 0 sec on 64.91.230.94
Failed Build #25 (Jan 30, 2014 9:34:36 AM)

  Building C object 
  CMakeFiles/cmTryCompileExec1338786878.dir/testCCompiler.c.o 

  /usr/lib/ccache/gcc -o 
  CMakeFiles/cmTryCompileExec1338786878.dir/testCCompiler.c.o -c 
  /tmp/buildd/ros-hydro-rosclean-1.10.9-0quantal-20140130-1736/obj-i686-linux-gnu/CMakeFiles/CMakeTmp/testCCompiler.c 

  ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/8/4: Permission 
  denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/pgAL_lzYC1I

Started 1 hr 3 min ago
Took 2 min 0 sec on 64.91.234.18
Failed Build #16 (Jan 30, 2014 9:34:52 AM)

[  1%] Building CXX object src/MiniCL/CMakeFiles/MiniCL.dir/MiniCL.o 
cd /tmp/buildd/ros-hydro-uwsim-bullet-2.79.0-3quantal-20140130-1736/obj-i686-linux-gnu/src/MiniCL && /usr/lib/ccache/c++   -DUSE_GRAPHICAL_BENCHMARK -fPIC -O3 -DNDEBUG -I/usr/local/include -I/tmp/buildd/ros-hydro-uwsim-bullet-2.79.0-3quantal-20140130-1736/src    -o CMakeFiles/MiniCL.dir/MiniCL.o -c /tmp/buildd/ros-hydro-uwsim-bullet-2.79.0-3quantal-20140130-1736/src/MiniCL/MiniCL.cpp 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/2/9: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/Zd5VDdxL3nM

Started 1 hr 3 min ago
Took 1 min 55 sec on 64.91.234.22
Failed Build #16 (Jan 30, 2014 9:35:27 AM)

[ 93%] Building CXX object sensor/sensor_odometry/CMakeFiles/sensor_odometry.dir/odometryreading.cpp.o 
cd /tmp/buildd/ros-hydro-openslam-gmapping-0.1.0-1quantal-20140130-1737/obj-i686-linux-gnu/sensor/sensor_odometry && /usr/lib/ccache/c++   -Dsensor_odometry_EXPORTS -fPIC -I/tmp/buildd/ros-hydro-openslam-gmapping-0.1.0-1quantal-20140130-1737/include    -o CMakeFiles/sensor_odometry.dir/odometryreading.cpp.o -c /tmp/buildd/ros-hydro-openslam-gmapping-0.1.0-1quantal-20140130-1737/sensor/sensor_odometry/odometryreading.cpp 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/8/2: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/LRc80Y_B86s

Started 1 hr 14 min ago
Took 1 min 11 sec on 64.91.234.20
Failed Build #21 (Jan 30, 2014 9:37:07 AM)

[  1%] Building CXX object src/CMakeFiles/orocos-bfl.dir/wrappers/matrix/matrix_BOOST.o 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/3/7: Permission denied

https://groups.google.com/forum/#!topic/ros-buildfarm-release/YMCTkqccQEQ

Started 1 hr 8 min ago
Took 2 min 14 sec on 64.91.234.25
Failed Build #28 (Jan 30, 2014 9:36:16 AM)

[ 24%] Building CXX object libstage/CMakeFiles/stage.dir/model_blobfinder.o 
cd /tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/obj-i686-linux-gnu/libstage && /usr/lib/ccache/c++   -Dstage_EXPORTS -O2 -DNDEBUG -Wall   -fPIC -I/tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/. -I/tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/libstage -I/tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/replace -I/usr/lib/fltk -I/tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/obj-i686-linux-gnu    -o CMakeFiles/stage.dir/model_blobfinder.o -c /tmp/buildd/ros-hydro-stage-4.1.1-5quantal-20140130-1738/libstage/model_blobfinder.cc 
ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/8/f: Permission denied
tfoote commented 10 years ago

I checked with our hosting and there are no known issues with the machines or the hosting.

tfoote commented 10 years ago

It looks like we've had about 71 instances of this error starting last March : https://groups.google.com/forum/#!searchin/ros-buildfarm-release/FATAL$3A$20%22Failed$20to$20create%22%7Csort:date

They seem to come at random, sometimes singally, or sometimes 9 or 10 at once. (Presumably when the buildfarm is running heavily.)

tfoote commented 10 years ago

Just observed again: 

CMake Error at /usr/share/cmake-2.8/Modules/CMakeTestCCompiler.cmake:61 (message):
  The C compiler "/usr/lib/ccache/cc" is not able to compile a simple test
  program.

  It fails with the following output:

   Change Dir: /tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp

  Run Build Command:/usr/bin/make "cmTryCompileExec136555328/fast"

  make[2]: Entering directory
  `/tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp'

  /usr/bin/make -f CMakeFiles/cmTryCompileExec136555328.dir/build.make
  CMakeFiles/cmTryCompileExec136555328.dir/build

  make[3]: Entering directory
  `/tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp'

  /usr/bin/cmake -E cmake_progress_report
  /tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp/CMakeFiles
  1

  Building C object
  CMakeFiles/cmTryCompileExec136555328.dir/testCCompiler.c.o

  /usr/lib/ccache/cc -g -O2 -fstack-protector --param=ssp-buffer-size=4
  -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -o
  CMakeFiles/cmTryCompileExec136555328.dir/testCCompiler.c.o -c
  /tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp/testCCompiler.c

  ccache: FATAL: Failed to create /var/cache/pbuilder/ccache/a/a: Permission
  denied

  make[3]: Leaving directory
  `/tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp'

  make[3]: *** [CMakeFiles/cmTryCompileExec136555328.dir/testCCompiler.c.o]
  Error 1

  make[2]: Leaving directory
  `/tmp/buildd/ros-indigo-catkin-0.6.5-0saucy-20140525-2011/obj-x86_64-linux-gnu/CMakeFiles/CMakeTmp'

  make[2]: *** [cmTryCompileExec136555328/fast] Error 2
dirk-thomas commented 9 years ago

We won't address this on the current farm. The upcoming new buildfarm does not use ccache anyway.