search

ros-infrastructure / ros_buildfarm

ROS buildfarm based on Docker
Apache License 2.0
81 stars 96 forks source link

race condition on testing repo sync vs doc jobs #203

Closed tfoote closed 8 years ago

tfoote commented 8 years ago

When the testing repo sync happens it can break doc jobs, and probably devel jobs too. This is an inherent race condition due to apt-get not supporting atomic operations. I'm not sure there's much we can do besides make the uploads more atomic. And to do that we'd probably need to change to a new repository hosting software.

We can probably close this as won't fix. @dirk-thomas thoughts?

The sync was at the same time: http://build.ros.org/job/Irel_sync-packages-to-testing_trusty_amd64/50/

Example failure: http://build.ros.org/job/Idoc__staubli_experimental__ubuntu_trusty_amd64/7/console

controller'
00:06:28.216 Reading package lists...
00:06:28.410 Building dependency tree...
00:06:28.411 Reading state information...
00:06:28.450 Starting pkgProblemResolver with broken count: 0
00:06:28.474 Starting 2 pkgProblemResolver with broken count: 0
00:06:28.475 Done
00:06:28.498 The following NEW packages will be installed:
00:06:28.498   ros-indigo-joint-state-controller
00:06:28.681 0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
00:06:28.681 Need to get 34.0 kB of archives.
00:06:28.681 After this operation, 179 kB of additional disk space will be used.
00:06:28.681 Err http://repositories.ros.org/ubuntu/testing/ trusty/main ros-indigo-joint-state-controller amd64 0.9.2-0trusty-20160110-091510-0800
00:06:28.681   404  Not Found
00:06:28.684 E: Failed to fetch http://repositories.ros.org/ubuntu/testing/pool/main/r/ros-indigo-joint-state-controller/ros-indigo-joint-state-controller_0.9.2-0trusty-20160110-091510-0800_amd64.deb  404  Not Found
00:06:28.684 
00:06:28.684 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
00:06:28.684 'apt-get install' failed and likely requires 'apt-get update' to run again
00:06:28.684 Invoking 'apt-get update'
00:06:29.027 Get:1 http://repositories.ros.org trusty InRelease [4,026 B]

http://build.ros.org/job/Idoc__jsk_visualization__ubuntu_trusty_amd64/11/console

00:10:01.661  ---> Running in 536923e04547
00:10:01.873 ros-indigo-catkin: 0.6.16-0trusty-20160109-081431-0800
00:10:01.915 Invoking 'apt-get update'
00:10:02.507 Get:1 http://repositories.ros.org trusty InRelease [4,026 B]
00:10:02.818 Ign http://archive.ubuntu.com trusty InRelease
00:10:02.930 Get:2 http://repositories.ros.org trusty/main amd64 Packages [229 kB]
00:10:03.181 Hit http://archive.ubuntu.com trusty-updates InRelease
00:10:03.181 Hit http://archive.ubuntu.com trusty-security InRelease
00:10:03.337 Hit http://archive.ubuntu.com trusty Release.gpg
00:10:03.495 Hit http://archive.ubuntu.com trusty-updates/main Sources
00:10:03.650 Hit http://archive.ubuntu.com trusty-updates/restricted Sources
00:10:03.806 Hit http://archive.ubuntu.com trusty-updates/universe Sources
00:10:03.964 Hit http://archive.ubuntu.com trusty-updates/main amd64 Packages
00:10:04.119 Hit http://archive.ubuntu.com trusty-updates/restricted amd64 Packages
00:10:04.275 Hit http://archive.ubuntu.com trusty-updates/universe amd64 Packages
00:10:04.431 Hit http://archive.ubuntu.com trusty Release
00:10:04.606 Hit http://archive.ubuntu.com trusty-security/main Sources
00:10:04.746 Hit http://archive.ubuntu.com trusty-security/restricted Sources
00:10:04.900 Hit http://archive.ubuntu.com trusty-security/universe Sources
00:10:05.080 Hit http://archive.ubuntu.com trusty-security/multiverse Sources
00:10:05.217 Hit http://archive.ubuntu.com trusty-security/main amd64 Packages
00:10:05.367 Hit http://archive.ubuntu.com trusty-security/restricted amd64 Packages
00:10:05.524 Hit http://archive.ubuntu.com trusty-security/universe amd64 Packages
00:10:05.683 Hit http://archive.ubuntu.com trusty-security/multiverse amd64 Packages
00:10:05.844 Hit http://archive.ubuntu.com trusty/main Sources
00:10:05.993 Hit http://archive.ubuntu.com trusty/restricted Sources
00:10:06.159 Hit http://archive.ubuntu.com trusty/universe Sources
00:10:06.314 Hit http://archive.ubuntu.com trusty/main amd64 Packages
00:10:06.469 Hit http://archive.ubuntu.com trusty/restricted amd64 Packages
00:10:06.719 Hit http://archive.ubuntu.com trusty/universe amd64 Packages
00:10:08.199 Fetched 233 kB in 5s (41.8 kB/s)
00:10:09.814 Reading package lists...
00:10:09.932 W: Size of file /var/lib/apt/lists/repositories.ros.org_ubuntu_testing_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported 228863 667855
00:10:09.932 Invoking 'apt-get install -q -y -o Debug::pkgProblemResolver=yes ros-indigo-catkin'
00:10:11.727 Reading package lists...
00:10:12.070 Building dependency tree...
00:10:12.071 Reading state information...
00:10:12.135 E: Unable to locate package ros-indigo-catkin
00:10:12.441 Removing intermediate container 536923e04547
00:10:12.443 The command '/bin/sh -c echo "ros-indigo-catkin: 0.6.16-0trusty-20160109-081431-0800" && python3 -u /tmp/wrapper_scripts/apt-get.py update-and-install -q -y -o Debug::pkgProblemResolver=yes ros-indigo-catkin' returned a non-zero code: 100
00:10:12.447 Build step 'Execute shell' marked build as failure
00:10:12.449 [ssh-agent] Stopped.

Similar errors: http://build.ros.org/job/Idoc__patrolling_sim__ubuntu_trusty_amd64/12/console http://build.ros.org/job/Idoc__moveit_commander__ubuntu_trusty_amd64/7/console http://build.ros.org/job/Idoc__flir_ptu__ubuntu_trusty_amd64/7/console

dirk-thomas commented 8 years ago

The ROS build farm is already designed in a way that it gracefully handles those kind of inconsistencies of the apt repository state. Until know we haven't seen this specific apt error yet: Unable to locate package. It will be trivial to add it as a know error case to the apt-get wrapper script and retry the installation step: #204.