Runing external programs are triggering timeout errors

[atruskie]

A BawAudioTools::Exceptions::AudioToolTimedOutError occurred in background at 2021-10-11 05:13:24 UTC :

  External Program: status=;killed=true;pid=1020;time_out_sec=60.0;time_taken_sec=61.26842316496186;timed_out=true;exceptions=[];command=ffprobe -hide_banner -sexagesimal -print_format default -show_error -show_streams -show_format "/data/original_audio/5f/5fd53df3-8283-4582-ad4b-c125bec88ead_20171223-141838Z.wav"
        Standard output:
        Standard Error:

  /home/baw_web/baw-server/lib/gems/baw-audio-tools/lib/baw_audio_tools/run_external_program.rb:62:in `execute'
  /home/baw_web/baw-server/lib/gems/baw-audio-tools/lib/baw_audio_tools/audio_base.rb:119:in `info_ffmpeg'
  /home/baw_web/baw-server/lib/gems/baw-audio-tools/lib/baw_audio_tools/audio_base.rb:94:in `info'
  /home/baw_web/baw-server/lib/gems/baw-audio-tools/lib/baw_audio_tools/audio_base.rb:266:in `modify'
  /home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/media/work_helper.rb:58:in `create_audio_segment'
  /home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/media/audio_job.rb:45:in `make_media_request'
  /home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/media/audio_job.rb:18:in `block in perform'
  /usr/local/bundle/gems/semantic_logger-4.8.2/lib/semantic_logger/base.rb:372:in `measure_internal'
  /usr/local/bundle/gems/semantic_logger-4.8.2/lib/semantic_logger/base.rb:97:in `measure_info'
  /home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/media/audio_job.rb:15:in `perform'

• There a number of zombie processes on hosts where these errors are happening.

• It's occurring across multiple hosts and multiple workers.

• Subsequent tests runs of the command run fine and quickly

• STDOUT and STDERR are almost always empty

• stack trace seems to be consistent

• Could be a process runner deadlock?

• Could be archived files?