search

ros-infrastructure / buildfarm

Build scripts and notes for catkin debian build pipeline.
6 stars 15 forks source link

groovy retry logic needs to backoff #94

Closed tfoote closed 11 years ago

tfoote commented 11 years ago

The retry logic needs to not retry continuously if it's an actual failure not a spurious failure. Otherwise it spams a poor user who committed at the wrong time.

Yesterday it appears there was an issue with aptproxy access such that the apt-get update failed. 

W: Failed to fetch http://aptproxy.willowgarage.com/us.archive.ubuntu.com/ubuntu/dists/quantal-updates/universe/binary-i386/Packages.bz2  Hash Sum mismatch

We have a groovy script to retry, but the failure presisted. So the job ran every 3 minutes all night.

Here's the last 10 logged jobs. 

  #2730 Feb 23, 2013 7:07:58 AM 
0.3.3-0
  #2729 Feb 23, 2013 3:21:36 AM  
  #2728 Feb 23, 2013 3:01:54 AM  
  #2727 Feb 23, 2013 2:57:43 AM  
  #2726 Feb 23, 2013 2:53:23 AM  
  #2725 Feb 23, 2013 2:47:39 AM  
  #2724 Feb 23, 2013 2:45:26 AM  
  #2723 Feb 23, 2013 2:38:23 AM  
  #2722 Feb 23, 2013 2:35:30 AM  
  #2721 Feb 23, 2013 2:28:46 AM
dirk-thomas commented 11 years ago

I will look at that tomorrow.