Harvest: timeout while processing HarvestItem should schedule a retry

[atruskie]

We should catch External Program killed errors and schedule a retry.

Production log:

 #<HarvestItem:0x00007f7de7873888
  id: 75964,
  path: "harvest_9/TAHD_3321/Site01_9_20201023_060000_Rec.wav",
  status: "errored",
  info:
   #<BawWorkers::Jobs::Harvest::Info error="\nExternal Program: status=;killed=true;pid=985;time_out_sec=60.0;time_taken_sec=61.90320665703621;timed_out=true;exceptions=[];command=ffmpeg -hide_banner -loglevel repeat+verbose -nostdin -i \"/data/harvester_to_do/harvest_9/TAHD_3321/Site01_9_20201023_060000_Rec.wav\" -codec copy -f null -\n\tStandard output: \n\tStandard Error: [wav @ 0x55eebaf852c0] parser not found for codec pcm_s16le, packets or times may be invalid.\nGuessed Channel Layout for Input Stream #0.0 : mono\nInput #0, wav, from '/data/harvester_to_do/harvest_9/TAHD_3321/Site01_9_20201023_060000_Rec.wav':\n  Duration: 06:45:47.84, bitrate: 705 kb/s\n    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, mono, s16, 705 kb/s\nOutput #0, null, to 'pipe:':\n  Metadata:\n    encoder         : Lavf58.45.100\n    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, mono, s16, 705 kb/s\nStream mapping:\n  Stream #0:0 -> #0:0 (copy)\nsize=N/A time=00:10:20.11 bitrate=N/A speed=1.24e+03x    \rsize=N/A time=00:22:19.74 bitrate=N/A speed=1.34e+03x    \rsize=N/A time=00:34:54.30 bitrate=N/A speed=1.4e+03x    \rsize=N/A time=00:43:07.67 bitrate=N/A speed=1.29e+03x    \rsize=N/A time=00:54:08.98 bitrate=N/A speed=1.29e+03x    \rsize=N/A time=01:05:17.71 bitrate=N/A speed=1.3e+03x    \rsize=N/A time=01:12:42.05 bitrate=N/A speed=1.24e+03x    \rsize=N/A time=01:21:05.83 bitrate=N/A speed=1.21e+03x    \rsize=N/A time=01:27:06.95 bitrate=N/A speed=1.16e+03x    \rsize=N/A time=01:33:57.10 bitrate=N/A speed=1.12e+03x    \rsize=N/A time=01:42:34.26 bitrate=N/A speed=1.12e+03x    \rsize=N/A time=01:50:23.86 bitrate=N/A speed=1.1e+03x    \rsize=N/A time=02:02:02.31 bitrate=N/A speed=1.12e+03x    \rsize=N/A time=02:10:06.78 bitrate=N/A speed=1.11e+03x    \rsize=N/A time=02:19:09.19 bitrate=N/A speed=1.11e+03x    \rsize=N/A time=02:28:59.17 bitrate=N/A speed=1.11e+03x    \rsize=N/A time=02:35:37.44 bitrate=N/A speed=1.09e+03x    \rsize=N/A time=02:46:03.07 bitrate=N/A speed=1.1e+03x    \rsize=N/A time=02:58:25.14 bitrate=N/A speed=1.12e+03x    \rsize=N/A time=03:10:50.64 bitrate=N/A speed=1.14e+03x    \rsize=N/A time=03:20:42.10 bitrate=N/A speed=1.14e+03x    \rsize=N/A time=03:30:36.53 bitrate=N/A speed=1.14e+03x    \rsize=N/A time=03:40:33.93 bitrate=N/A speed=1.15e+03x    \rsize=N/A time=03:51:14.94 bitrate=N/A speed=1.15e+03x    \rsize=N/A time=04:03:15.18 bitrate=N/A speed=1.16e+03x    \rsize=N/A time=04:12:01.25 bitrate=N/A speed=1.15e+03x    \rsize=N/A time=04:24:33.57 bitrate=N/A speed=1.17e+03x    \rsize=N/A time=04:32:22.80 bitrate=N/A speed=1.16e+03x    \rsize=N/A time=04:32:24.29 bitrate=N/A speed= 857x    \rsize=N/A time=04:33:20.76 bitrate=N/A speed= 838x    \rsize=N/A time=04:34:33.58 bitrate=N/A speed= 820x    \rsize=N/A time=04:34:54.38 bitrate=N/A speed= 630x    \rsize=N/A time=04:35:18.16 bitrate=N/A speed= 619x    \rsize=N/A time=04:37:06.64 bitrate=N/A speed= 611x    \rsize=N/A time=04:45:43.80 bitrate=N/A speed= 619x    \rsize=N/A time=04:56:06.51 bitrate=N/A speed= 630x    \rsize=N/A time=05:05:28.20 bitrate=N/A speed= 639x    \rsize=N/A time=05:10:37.31 bitrate=N/A speed= 638x    \rsize=N/A time=05:15:15.20 bitrate=N/A speed= 637x    \rsize=N/A time=05:15:56.81 bitrate=N/A speed= 627x    \rsize=N/A time=05:17:18.55 bitrate=N/A speed= 516x    \rsize=N/A time=05:17:54.21 bitrate=N/A speed= 509x    \rsize=N/A time=05:18:32.85 bitrate=N/A speed= 503x    \rsize=N/A time=05:19:51.61 bitrate=N/A speed= 498x    \rsize=N/A time=05:20:06.47 bitrate=N/A speed= 456x    \rsize=N/A time=05:20:58.49 bitrate=N/A speed= 452x    \rsize=N/A time=05:21:08.89 bitrate=N/A speed= 441x    \rsize=N/A time=05:22:45.48 bitrate=N/A speed= 438x    \rsize=N/A time=05:23:59.79 bitrate=N/A speed= 435x    \rsize=N/A time=05:26:01.65 bitrate=N/A speed= 433x    \rsize=N/A time=05:37:11.87 bitrate=N/A speed= 443x    \rsize=N/A time=05:44:38.06 bitrate=N/A speed= 448x    \rsize=N/A time=05:55:21.16 bitrate=N/A speed= 457x    \rsize=N/A time=05:58:07.60 bitrate=N/A speed= 433x    \rsize=N/A time=05:58:40.30 bitrate=N/A speed= 406x    \rsize=N/A time=06:00:12.43 bitrate=N/A speed= 403x    \rsize=N/A time=06:01:07.42 bitrate=N/A speed= 401x    \rsize=N/A time=06:01:32.68 bitrate=N/A speed= 374x    \rsize=N/A time=06:02:27.67 bitrate=N/A speed= 371x    \r\n\n" fixes=[{"problems"=>{"FL001"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL005"=>{"status"=>"Fixed", "message"=>"RIFF length set to 2147479588 (was 2147479632). data length set to 2147479552 (was 2147479596)", "new_path"=>nil, "check_result"=>{"data"=>{"data_chunk"=>{"end"=>2147479640, "start"=>44, "length"=>2147479596, "out_of_bounds"=>true}, "riff_chunk"=>{"end"=>2147479640, "start"=>8, "length"=>2147479632, "out_of_bounds"=>true}, "format_chunk"=>{"end"=>36, "start"=>20, "length"=>16, "out_of_bounds"=>false}}, "status"=>"Affected", "message"=>"RIFF length and data length are incorrect", "severity"=>"Mild"}}, "FL008"=>{"status"=>"NoOperation", "message"=>"", "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL010"=>{"status"=>"NoOperation", "message"=>"Audio recording is not a FLAC file", "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"Audio recording is not a FLAC file", "severity"=>"None"}}}}, {"problems"=>{"FL001"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL005"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>nil, "severity"=>"None"}}, "FL008"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL010"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"Audio recording is not a FLAC file", "severity"=>"None"}}}}, {"problems"=>{"FL001"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL005"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>nil, "severity"=>"None"}}, "FL008"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"", "severity"=>"None"}}, "FL010"=>{"status"=>"NoOperation", "message"=>nil, "new_path"=>nil, "check_result"=>{"data"=>nil, "status"=>"Unaffected", "message"=>"Audio recording is not a FLAC file", "severity"=>"None"}}}}] file_info={:path=>"TAHD_3321", :notes=>{:relative_path=>"harvest_9/TAHD_3321/Site01_9_20201023_060000_Rec.wav"}, :prefix=>"Site01_9_", :suffix=>"_Rec", :site_id=>3321, :channels=>1, :extension=>"wav", :file_hash=>"SHA256::7fe76e14fe004647a0f9707d4eb8398f567787f3bae1fb671900a4f065b4611f", :file_name=>"Site01_9_20201023_060000_Rec.wav", :recursive=>true, :media_type=>"audio/wav", :utc_offset=>"+10:00", :access_time=>"2022-08-03T04:00:04.000+00:00", :change_time=>"2022-08-03T03:33:45.000+00:00", :uploader_id=>1058, :bit_rate_bps=>705600, :modified_time=>"2022-08-03T03:33:45.000+00:00", :recorded_date=>"2020-10-23T06:00:00.000+10:00", :duration_seconds=>24347.841, :data_length_bytes=>2147479596, :sample_rate_hertz=>44100, :recorded_date_local=>"2020-10-23T06:00:00.000+00:00"} validations=[]>,
  audio_recording_id: nil,
  uploader_id: 1058,
  created_at: Wed, 03 Aug 2022 03:33:35.031612000 UTC +00:00,
  updated_at: Wed, 03 Aug 2022 04:03:16.025932000 UTC +00:00,
  harvest_id: 9,
  deleted: false>,