In this scenario the ffprobe executable was missing. AP.exe correctly failed to execute and return an ERRORLEVEL of 52. This non-zero exit code was not detected and the job appears to have registered as successful. There is a job.success file, the action does not get moved to the failed queue, and the worker.log file thinks the run is successful as well.
Ensure the job fails is the executable script that is run fails.
AP.exe log:
2016-05-02T09:35:05.1706370+00:00 [1] INFO LoggedConsole - QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54)
Git branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5
Copyright QUT 2016
2016-05-02T09:35:05.1836340+00:00 [1] INFO LogFileOnly - Executable called with these arguments:
/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe audio2csv /source:/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav /config:/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config /tempdir:/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp /output:/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2
2016-05-02T09:35:05.7220910+00:00 [1] INFO LoggedConsole - # PROCESS LONG RECORDING
2016-05-02T09:35:05.7224820+00:00 [1] INFO LoggedConsole - # DATE AND TIME: 05/02/2016 09:35:05
2016-05-02T09:35:05.7225780+00:00 [1] INFO LoggedConsole - # Recording file: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav
2016-05-02T09:35:05.7228770+00:00 [1] INFO LoggedConsole - # Configuration file: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config
2016-05-02T09:35:05.7229810+00:00 [1] INFO LoggedConsole - # Output folder: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2
2016-05-02T09:35:05.7231240+00:00 [1] INFO LoggedConsole - # Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp
2016-05-02T09:35:05.8615460+00:00 [1] INFO AnalysisPrograms.AnalyseLongRecordings.AnalyseLongRecording - Minimum event threshold has been set to 0.2
2016-05-02T09:35:05.8811140+00:00 [1] INFO AnalysisBase.FileSegment - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00
2016-05-02T09:35:05.8815530+00:00 [1] WARN AnalysisPrograms.AnalyseLongRecordings.AnalyseLongRecording - Neither start nor end segment offsets provided. Therefore ignored
2016-05-02T09:35:06.0472510+00:00 [1] INFO AnalysisPrograms.Acoustic - Image tiling is enabled
2016-05-02T09:35:06.0507030+00:00 [1] INFO LoggedConsole - STARTING ANALYSIS ...
2016-05-02T09:35:06.0783340+00:00 [1] INFO LoggedConsole - FATAL ERROR:
Could not find exe: /usr/local/bin/ffprobe
2016-05-02T09:35:06.1169140+00:00 [1] INFO LoggedConsole - Usage: AnalysisPrograms <action> options
Global options:
-d -debug [switch] *
-n -nodebug [switch] *
-l -loglevel [logverbosity] *
-v -verbose [switch] *
-vv -vverbose [switch] *
Action:
audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.
-so -source file 1 The source audio file to operate on
-c -config file 2 The path to the config file
-o -output directory 3 A directory to write output to
-t -tempdir [directory] * A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).
-s -startoffset [double] * The start offset to start analyzing from (in seconds)
-e -endoffset [double] * The end offset to stop analyzing (in seconds)
-w -whenexitcopylog [switch] * If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.
-wh -whenexitcopyconfig [switch] * If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied
2016-05-02T09:35:06.1175700+00:00 [1] INFO LogFileOnly - ERRORLEVEL: 52
worker.log:
# Logfile created on 2016-05-02 09:35:04 +0000 by logger.rb/47272
2016-05-02T09:35:04.936+0000 [ INFO-BawWorkers::Analysis::Runner-005860] Executing mono "/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe" audio2csv /source:"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav" /config:"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config" /tempdir:"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp" /output:"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2".
2016-05-02T09:35:06.453+0000 [DEBUG-BawAudioTools::RunExternalProgram-005860] External Program: status=52;killed=false;pid=5864;time_out_sec=7200;time_taken_sec=1.516216453164816;timed_out=false;exceptions=[];command=mono "/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe" audio2csv /source:"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav" /config:"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config" /tempdir:"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp" /output:"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2"
Standard output: QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54)
Git branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5
Copyright QUT 2016
# PROCESS LONG RECORDING
# DATE AND TIME: 05/02/2016 09:35:05
# Recording file: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav
# Configuration file: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config
# Output folder: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2
# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp
[2016-05-02T09:35:05.8615460+00:00] INFO - Minimum event threshold has been set to 0.2
[2016-05-02T09:35:05.8811140+00:00] INFO - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00
[2016-05-02T09:35:05.8815530+00:00] WARN - Neither start nor end segment offsets provided. Therefore ignored
[2016-05-02T09:35:06.0472510+00:00] INFO - Image tiling is enabled
STARTING ANALYSIS ...
FATAL ERROR:
Could not find exe: /usr/local/bin/ffprobe
Usage: AnalysisPrograms <action> options
Global options:
-d -debug [switch] *
-n -nodebug [switch] *
-l -loglevel [logverbosity] *
-v -verbose [switch] *
-vv -vverbose [switch] *
Action:
audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.
-so -source file 1 The source audio file to operate on
-c -config file 2 The path to the config file
-o -output directory 3 A directory to write output to
-t -tempdir [directory] * A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk).
-s -startoffset [double] * The start offset to start analyzing from (in seconds)
-e -endoffset [double] * The end offset to stop analyzing (in seconds)
-w -whenexitcopylog [switch] * If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory.
-wh -whenexitcopyconfig [switch] * If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied
Standard Error:
2016-05-02T09:35:06.453+0000 [DEBUG-BawWorkers::Analysis::Runner-005860] Successfully executed {:command=>"mono \"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\" audio2csv /source:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\" /config:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\" /tempdir:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\" /output:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\"", :stdout=>"QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\nCopyright QUT 2016\n# PROCESS LONG RECORDING\n# DATE AND TIME: 05/02/2016 09:35:05\n# Recording file: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\n# Configuration file: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\n# Output folder: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\n# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\n[2016-05-02T09:35:05.8615460+00:00] INFO - Minimum event threshold has been set to 0.2\n[2016-05-02T09:35:05.8811140+00:00] INFO - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\n[2016-05-02T09:35:05.8815530+00:00] WARN - Neither start nor end segment offsets provided. Therefore ignored\n[2016-05-02T09:35:06.0472510+00:00] INFO - Image tiling is enabled\nSTARTING ANALYSIS ...\nFATAL ERROR:\n\tCould not find exe: /usr/local/bin/ffprobe\nUsage: AnalysisPrograms <action> options\n\nGlobal options:\n -d -debug [switch] * \n -n -nodebug [switch] * \n -l -loglevel [logverbosity] * \n -v -verbose [switch] * \n -vv -vverbose [switch] * \nAction: \n audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\n -so -source file 1 The source audio file to operate on \n -c -config file 2 The path to the config file \n -o -output directory 3 A directory to write output to \n -t -tempdir [directory] * A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk). \n -s -startoffset [double] * The start offset to start analyzing from (in seconds) \n -e -endoffset [double] * The end offset to stop analyzing (in seconds) \n -w -whenexitcopylog [switch] * If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory. \n -wh -whenexitcopyconfig [switch] * If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied \n\n", :stderr=>"", :time_taken=>1.516216453164816, :exit_code=>52, :execute_msg=>"External Program: status=52;killed=false;pid=5864;time_out_sec=7200;time_taken_sec=1.516216453164816;timed_out=false;exceptions=[];command=mono \"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/programs/AnalysisPrograms/AnalysisPrograms.exe\" audio2csv /source:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\" /config:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\" /tempdir:\"/mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\" /output:\"/home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\"\n\tStandard output: QUT Bioacoustic Analysis Program - version 15.10.3309.0 (RELEASE build, 10/30/2015 06:54) \nGit branch-version: master-7cd0fa5e863158369c8d7043266a6f36342c1cf5\nCopyright QUT 2016\n# PROCESS LONG RECORDING\n# DATE AND TIME: 05/02/2016 09:35:05\n# Recording file: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/original_audio/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2_20150906-210000Z.wav\n# Configuration file: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/run.config\n# Output folder: /home/ubuntu/bioacoustics/qcif_storage_nfs/data_prod/analysis_results/system/54/54a9f5fb-9f34-4731-8fa7-2b0405657df2\n# Temp File Directory: /mnt/workers/production/01/runs/system_336487_2016_05_02t09_35_04z/temp\n[2016-05-02T09:35:05.8615460+00:00] INFO - Minimum event threshold has been set to 0.2\n[2016-05-02T09:35:05.8811140+00:00] INFO - Parsed file start date as 2015-09-06T21:00:00.0000000+00:00\n[2016-05-02T09:35:05.8815530+00:00] WARN - Neither start nor end segment offsets provided. Therefore ignored\n[2016-05-02T09:35:06.0472510+00:00] INFO - Image tiling is enabled\nSTARTING ANALYSIS ...\nFATAL ERROR:\n\tCould not find exe: /usr/local/bin/ffprobe\nUsage: AnalysisPrograms <action> options\n\nGlobal options:\n -d -debug [switch] * \n -n -nodebug [switch] * \n -l -loglevel [logverbosity] * \n -v -verbose [switch] * \n -vv -vverbose [switch] * \nAction: \n audio2csv - Calls AnalyseLongRecording.Execute(): Outputs acoustic indices and LD false-colour spectrograms.\n -so -source file 1 The source audio file to operate on \n -c -config file 2 The path to the config file \n -o -output directory 3 A directory to write output to \n -t -tempdir [directory] * A TEMP directory where cut files will be stored. Use this option for efficiency (e.g. write to a RAM Disk). \n -s -startoffset [double] * The start offset to start analyzing from (in seconds) \n -e -endoffset [double] * The end offset to stop analyzing (in seconds) \n -w -whenexitcopylog [switch] * If true, attempts to copy the executable's log file to output directory. If it can't determine an output directory, it copies to the working directory. \n -wh -whenexitcopyconfig [switch] * If true, attempts to copy the executable's config file to output directory. If it can't determine an output directory, it copies to the working directory. If it can't find a config file, nothing is copied \n\n\n\tStandard Error: "}.
In this scenario the
ffprobeexecutable was missing. AP.exe correctly failed to execute and return anERRORLEVELof52. This non-zero exit code was not detected and the job appears to have registered as successful. There is ajob.successfile, the action does not get moved to the failed queue, and theworker.logfile thinks the run is successful as well.Ensure the job fails is the executable script that is run fails.
AP.exe log:
worker.log: