Undetected Executables' path errors

azzaea commented 6 years ago

It seems the pipeline is unable to tell when the executable of one of the programs being called (say bwa) is non-existing. These checks were already in place in the bash version of the pipeline, but we skipped this part while building the Swift/T equivalent.

The tricky part is that the error messages resulting from erroneous usage are silently dropped, without alarming the user of the problem. To be more specific, the error message does get printed, but in place of the expected output; and since nothing gets produced (i.e., no exit code from the app function), the pipeline simply hangs, and the user stays in the dark.

Practical example when bwa path is erroneously specified in the runfile:

$ cat tmp_analysis/align/H3A_NextGen_assessment_set3.noDedups.sam
CAUGHT ERROR:^M
^M
error: c::sync_exec: Error executing command /home/apps/bwa/bwa-0.7.15: No such file or directory^M
^M
^M
    invoked from within^M
"c::sync_exec $stdin_src $stdout_dst $stderr_dst $cmd {*}$args"^M
    (procedure "turbine::exec_external" line 19)^M
    invoked from within^M
"turbine::exec_external ${v:bwaexe} [ dict create "stdout" ${v:__filename:output} "stderr" ${v:__filename:outLog} ] "mem" "-M" {*}${v:bwamemparams} "-t..."^M
    (procedure "bwa_mem-app-leaf1" line 35)^M
    invoked from within^M
"bwa_mem-app-leaf1 152.0 152.1 150.0 157 163 164 165 0 1 H3A_NextGen_assessment_set3 {H3A_NextGen_assessment_set3       ALL     BWAMEM start    1506715523^M
} {file 1..."^M
Turbine worker task error in: bwa_mem-app-leaf1 152.0 152.1 150.0 157 163 164 165 0 1 H3A_NextGen_assessment_set3 {H3A_NextGen_assessment_set3  ALL     BWAMEM start    1506715523^M
} {file 160 is_mapped 1} {file 150 is_mapped 1} 140 153 {file 161 is_mapped 1}^M
    invoked from within^M
"c::worker_loop $WORK_TYPE($mode) $keyword_args"^M
    (procedure "standard_worker" line 27)^M
    invoked from within^M
"standard_worker $rules $startup_cmd "^M
    (procedure "enter_mode_unchecked" line 5)^M
    invoked from within^M
"enter_mode_unchecked $rules $startup_cmd"^M
    (procedure "enter_mode" line 5)^M
    invoked from within^M
"enter_mode $rules $startup_cmd "^M
ADLB: ADLB_Abort(1) calling MPI_Abort(1)

azzaea commented 6 years ago

The last push checks that:

The executable being referred actually exists.
That it is actually a file

j-woz commented 6 years ago

Hi- where is the fix for this? Thanks

azzaea commented 6 years ago

I needed the workflow to kind of fail safely should the user erroneously specified the path to one of the executables. By safely, I mean that the error message should direct the user to which parameter in the runfile had a problem with path specification. Code is below:

// Modifying the pure swift/t function to accept strings as well (since for tcl everything is a string, this is straight forward) - I need to call this in my exec_check function to guarantee the 'fail-safe' aspect
@pure
(string t) file_type(string f)
"turbine" "0.0.2"
[ "set <<t>> [ file type [ lindex <<f>> 0 ] ]" ];

(boolean exec_ok) exec_check (string exec, string parameter){               // Checking function definition
        file_exists(exec) =>                                                // This merely tests 'exec' is an entity in the file system, so not sufficient alone
        assert(string_count(file_type(exec), "file", 0, -1) ==1,           // If exec is not a file, exit and give warning
        strcat("The executable: \n\t", exec,
            "\n Referred to by the parameter: \n\t", parameter, 
            "is not properly specified in your runfile!"));
        exec_ok = true;
}

Is there a better way to do/think about this?

ncsa / Swift-T-Variant-Calling

Undetected Executables' path errors #24