search

jyyulab / SJARACNe

Scalable Tool for Gene Network Reverse Engineering
Other
22 stars 16 forks source link

The argument [adjmat_dir] for step [consensus] is required #21

Closed adamdingliang closed 2 years ago

adamdingliang commented 5 years ago

nodecn232 rprod SJARACNe$ sjaracne lsf -j ./SJARACNe/config/config_cwlexec.json -e ./test_data/inputs/BRCA100.exp -g ./test_data/inputs/tf.txt -n 2 -o ./results INFO:root:results/sjaracne_workflow.yml INFO:root:exp_file: class: File path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/test_data/inputs/BRCA100.exp probe_file: class: File path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/test_data/inputs/tf.txt p_value_consensus: 1e-05 p_value_bootstrap: 1e-07 depth: 40 aracne_config_dir: class: Directory path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/config bootstrap_num: 2 final_out_dir_name: results INFO:root:cwlexec -pe PATH -c ./SJARACNe/config/config_cwlexec.json --outdir ./results /research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/cwl/sjaracne_workflow.cwl results/sjaracne_workflow.yml [16:41:21.584] INFO - Workflow ID: 09f6e18f-d40a-41c1-a8ed-fb124d14ca90 [16:41:21.585] INFO - Name: sjaracne_workflow [16:41:21.585] INFO - Description file path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/cwl/sjaracne_workflow.cwl [16:41:21.585] INFO - Input settings file path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/results/sjaracne_workflow.yml [16:41:21.585] INFO - Execution configuration path: /research/rgs01/home/clusterHome/lding/Git/SJARACNe/./SJARACNe/config/config_cwlexec.json [16:41:21.585] INFO - Output directory: /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 [16:41:21.585] INFO - Work directory: /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 [16:41:21.585] INFO - Workflow "sjaracne_workflow" started to execute. [16:41:21.592] INFO - Started job (validate_files) with bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/validate_files \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=100MB] \ QC_input.py -e BRCA100.exp -g tf.txt [16:41:21.595] INFO - Started job (ch_ending_exp) with bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_exp \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=100MB] \ ch_line_ending.py -i BRCA100.exp [16:41:21.605] INFO - Started job (ch_ending_probe) with bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=100MB] \ ch_line_ending.py -i tf.txt [16:41:21.619] INFO - Pre-submitted job (bootstrap) with a placeholder command: bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=9999MB] \ -H \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/bootstrap [16:41:21.622] INFO - Pre-submitted job (copy_to_dir) with a placeholder command: bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/copy_to_dir \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=100MB] \ -H \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/copy_to_dir/copy_to_dir [16:41:21.623] INFO - Pre-submitted job (consensus) with a placeholder command: bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/consensus \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=20000MB] \ -H \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/consensus/consensus [16:41:22.221] INFO - Job (ch_ending_exp) was submitted. Job <89794467> is submitted to queue . [16:41:22.261] INFO - Job (consensus) was submitted. Job <89794469> is submitted to queue . [16:41:22.264] INFO - Job (validate_files) was submitted. Job <89794471> is submitted to queue . [16:41:22.265] INFO - Job (create_adjmat_names) was submitted. Job <89794474> is submitted to queue . [16:41:22.267] INFO - Job (create_seeds) was submitted. Job <89794470> is submitted to queue . [16:41:22.274] INFO - Job (bootstrap) was submitted. Job <89794472> is submitted to queue . [16:41:22.275] INFO - Job (ch_ending_probe) was submitted. Job <89794473> is submitted to queue . [16:41:22.275] INFO - Job (copy_to_dir) was submitted. Job <89794468> is submitted to queue . [16:41:22.279] INFO - Started to wait for jobs by bwait \ -w \ done(89794471) [16:41:22.294] INFO - Started to wait for jobs by bwait \ -w \ done(89794470) && done(89794474) && done(89794467) && done(89794473) [16:41:25.082] INFO - The job (validate_files) <89794471> is done with stdout from LSF: Job <QC_input.py -e BRCA100.exp -g tf.txt> was submitted from host by user in cluster at Thu Nov 7 16:41:22 2019. Job was executed on host(s) , in queue , as user in cluster at Thu Nov 7 16:41:23 2019. </home/lding> was used as the home directory. </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/validate_files> was used as the working directory. Started at Thu Nov 7 16:41:23 2019. Terminated at Thu Nov 7 16:41:24 2019. Results reported at Thu Nov 7 16:41:24 2019.

LSBATCH: User input

QC_input.py -e BRCA100.exp -g tf.txt

Successfully completed. Resource usage summary: CPU time : 1.13 sec. Max Memory : 15 MB Average Memory : 15.00 MB Total Requested Memory : 100.00 MB Delta Memory : 85.00 MB Max Swap : - Max Processes : 3 Max Threads : 4 Run time : 2 sec. Turnaround time : 2 sec. The output (if any) follows:

[16:41:25.538] INFO - The job (ch_ending_exp) <89794467> is done with stdout from LSF: Job <ch_line_ending.py -i BRCA100.exp> was submitted from host by user in cluster at Thu Nov 7 16:41:22 2019. Job was executed on host(s) , in queue , as user in cluster at Thu Nov 7 16:41:22 2019. </home/lding> was used as the home directory. </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_exp> was used as the working directory. Started at Thu Nov 7 16:41:22 2019. Terminated at Thu Nov 7 16:41:24 2019. Results reported at Thu Nov 7 16:41:24 2019.

LSBATCH: User input

ch_line_ending.py -i BRCA100.exp

Successfully completed. Resource usage summary: CPU time : 0.12 sec. Max Memory : 14 MB Average Memory : 13.00 MB Total Requested Memory : 100.00 MB Delta Memory : 86.00 MB Max Swap : - Max Processes : 3 Max Threads : 4 Run time : 1 sec. Turnaround time : 2 sec. The output (if any) follows:

[16:41:25.560] INFO - The job (ch_ending_probe) <89794473> is done with stdout from LSF: Job <ch_line_ending.py -i tf.txt> was submitted from host by user in cluster at Thu Nov 7 16:41:22 2019. Job was executed on host(s) , in queue , as user in cluster at Thu Nov 7 16:41:23 2019. </home/lding> was used as the home directory. </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe> was used as the working directory. Started at Thu Nov 7 16:41:23 2019. Terminated at Thu Nov 7 16:41:24 2019. Results reported at Thu Nov 7 16:41:24 2019.

LSBATCH: User input

ch_line_ending.py -i tf.txt

Successfully completed. Resource usage summary: CPU time : 0.10 sec. Max Memory : - Average Memory : - Total Requested Memory : 100.00 MB Delta Memory : - Max Swap : - Max Processes : - Max Threads : - Run time : 2 sec. Turnaround time : 2 sec. The output (if any) follows:

[16:41:25.571] INFO - The step (bootstrap) scatter of 2 jobs by dotproduct. [16:41:25.571] INFO - Started job (bootstrap_1) with bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/scatter1 \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=9999MB] \ sjaracne.exe -i /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_exp/BRCA100.exp -l /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe/tf.txt -s /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe/tf.txt -p 1.0E-7 -e 0 -a adaptive_partitioning -r 1 -H /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/config -N 40 -o TF_run_000.adj -S 0 [16:41:25.571] INFO - Started job (bootstrap_2) with bsub \ -cwd \ /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/scatter2 \ -o \ %J_out \ -e \ %J_err \ -env \ PATH,TMPDIR=/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90 \ -q \ compbio \ -P \ SJARACNe \ -R \ rusage[mem=9999MB] \ sjaracne.exe -i /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_exp/BRCA100.exp -l /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe/tf.txt -s /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_probe/tf.txt -p 1.0E-7 -e 0 -a adaptive_partitioning -r 1 -H /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/config -N 40 -o TF_run_001.adj -S 1 [16:41:25.686] INFO - Job (bootstrap_1) was submitted. Job <89794475> is submitted to queue . [16:41:25.686] INFO - Job (bootstrap_2) was submitted. Job <89794476> is submitted to queue . [16:41:25.687] INFO - Started to wait for jobs by bwait \ -w \ done(89794475) && done(89794476)

[17:00:08.422] INFO - Fill out the scatter gather result in the script /home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/bootstrap [17:00:08.425] INFO - Resuming job (bootstrap) <89794472> with bresume \ 89794472 [17:00:09.708] INFO - The job (bootstrap) <89794472> is done with stdout from LSF: Job </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/bootstrap> was submitted from host by user in cluster at Thu Nov 7 16:41:22 2019. Job was executed on host(s) , in queue , as user in cluster at Thu Nov 7 17:00:09 2019. </home/lding> was used as the home directory. </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap> was used as the working directory. Started at Thu Nov 7 17:00:09 2019. Terminated at Thu Nov 7 17:00:09 2019. Results reported at Thu Nov 7 17:00:09 2019.

LSBATCH: User input

/home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/bootstrap/bootstrap

Successfully completed. Resource usage summary: CPU time : 0.04 sec. Max Memory : - Average Memory : - Total Requested Memory : 9999.00 MB Delta Memory : - Max Swap : - Max Processes : 1 Max Threads : 1 Run time : 1 sec. Turnaround time : 1127 sec. The output (if any) follows:

[17:00:09.759] INFO - Resuming job (copy_to_dir) <89794468> with bresume \ 89794468 [17:00:09.783] INFO - Started to wait for jobs by bwait \ -w \ done(89794467) && done(89794468) [17:00:10.936] INFO - The job (ch_ending_exp) <89794467> is done with stdout from LSF: Job <ch_line_ending.py -i BRCA100.exp> was submitted from host by user in cluster at Thu Nov 7 16:41:22 2019. Job was executed on host(s) , in queue , as user in cluster at Thu Nov 7 16:41:22 2019. </home/lding> was used as the home directory. </home/lding/cwl-workdir/09f6e18f-d40a-41c1-a8ed-fb124d14ca90/ch_ending_exp> was used as the working directory. Started at Thu Nov 7 16:41:22 2019. Terminated at Thu Nov 7 16:41:24 2019. Results reported at Thu Nov 7 16:41:24 2019.

LSBATCH: User input

ch_line_ending.py -i BRCA100.exp

Successfully completed. Resource usage summary: CPU time : 0.12 sec. Max Memory : 14 MB Average Memory : 13.00 MB Total Requested Memory : 100.00 MB Delta Memory : 86.00 MB Max Swap : - Max Processes : 3 Max Threads : 4 Run time : 1 sec. Turnaround time : 2 sec. The output (if any) follows:

[17:00:10.993] ERROR - Failed to wait for job consensus <89794469>, The argument [adjmat_dir] for step [consensus] is required. [17:00:10.993] ERROR - The workflow (sjaracne_workflow) exited with <255>. [17:00:10.993] WARN - killing waiting job (consensus) <89794469>. Traceback (most recent call last): File "/home/lding/.conda/envs/py361_test/bin/sjaracne", line 11, in load_entry_point('SJARACNe==0.2.0', 'console_scripts', 'sjaracne')() File "/research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/sjaracne.py", line 99, in main run_shell_command_call(cmd) File "/research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/sjaracne.py", line 110, in run_shell_command_call subprocess.check_call(cmd_to_exec) File "/home/lding/.conda/envs/py361_test/lib/python3.6/subprocess.py", line 291, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['cwlexec', '-pe', 'PATH', '-c', './SJARACNe/config/config_cwlexec.json', '--outdir', './results', '/research/rgs01/home/clusterHome/lding/Git/SJARACNe/SJARACNe/cwl/sjaracne_workflow.cwl', 'results/sjaracne_workflow.yml']' returned non-zero exit status 255.