./results/.db/run: No such file or directory

[quzhouxiachuan]

Hi,

I set up my pipeline and installed all the dependency correctly, but it popped up error when I try "run" to run _01 a-aglign command: [ydw529@qnode4144 _01a-align]$ ./run === Operation = align ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES cat: ./results/.db/run: No such file or directory cat: ./results/.db/run: No such file or directory

I then tried ./run.dry, and it shows the follow message:

_[ydw529@qnode4144 projecttest]$ ./run.dry Validating sample sheet... PIPELINE STARTING: Tue May 24 13:50:33 CDT 2016 === Operation = align ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = filter ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = filter-stats ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = tracks ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = matrix-filtered ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = matrix-prep ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = matrix-ic ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = matrix-hicnorm ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = matrix-stats ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = compare-matrices ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = compare-matrices-stats ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = boundary-scores ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = boundary-scores-pca ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = domains ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = domains-stats ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = compare-boundaries ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = compare-boundaries-stats ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = hicplotter ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = interactions ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES === Operation = annotations ============= **Warning: .db directory exists, removing! Error: parameter file list is empty!**

What do you think could be the problem here?

[stevekm]

I just checked and was able to reproduce the same error by trying to run a new analysis pipeline without setting any input files. Have you checked to make sure your inputs directory is set correctly?

[quzhouxiachuan]

My inputs look like this:

---

_lrwxrwxrwx 1 ydw529 b1034 7 May 16 10:22 code -> ../code lrwxrwxrwx 1 ydw529 b1034 7 May 16 10:22 data -> ../data drwxrwsr-x 3 ydw529 b1034 32768 May 17 12:14 fastq lrwxrwxrwx 1 ydw529 b1034 12 May 16 10:22 genomes -> data/genomes drwxrwsr-x 2 ydw529 b1034 32768 May 25 18:03 params -rw-rw-r-- 1 ydw529 b1034 483 May 16 10:22 README lrwxrwxrwx 1 ydw529 b1034 5 May 16 17:15 results -> fastq -rw-rw---- 1 ydw529 b1034 157 May 25 18:04 sample-sheet.tv [ydw529@quser12 inputs]$ cd fastq [ydw529@quser12 fastq]$ ll total 32 drwxrwsr-x 2 ydw529 b1034 32768 May 24 11:07 ESC-Hindiii-R1 [ydw529@quser12 fastq]$ cd ESC-Hindiii-R1/ [ydw529@quser12 ESC-Hindiii-R1]$ ll total 50540544 -rw-rw-r-- 1 ydw529 b1034 26858225664 May 24 11:08 ESC-Hindiii_R1.fastq -rw-rw-r-- 1 ydw529 b1034 24895291392 May 17 12:24 ESC-HindiiiR2.fastq

Here is the sample sheet: _sample group fastq-r1 fastq-r2 genome enzyme ESC-Hindiii-R1 ESC-Hindiii ESC-Hindiii-R1/ESC-Hindiii_R1.fastq ESC-Hindiii-R1/ESC-HindiiiR2.fastq hg19 HindIII

Also, when I used git clone to set up the pipeline, the link of inputs/genomes is broken. Would you be able to provide files under genomes folder?

[tsirigos]

you have a file named "sample-sheet.tv", but it should be "tsv".

[quzhouxiachuan]

Hi

It is tsv in my terminal, I probably accidentally deleted "s" when I was copying pasting it. Also, we use a different scheduler instead of OGS/ Grid Engine 2011.11, could that be the problem?

[quzhouxiachuan]

I also customized my param.tcsh as the following

_#!/bin/tcsh

load basic tools

module load python module load R module load matlab module load gcc/4.8.3 module load gsl/1.16-gcc4.8.3 module load boost module load bowtie2/2.2.6 module use /projects/b1034/yd/HiC/hic-bench/modules module load bedtools/2.25 module load armatus/2.0 module load ghmm/0.9 module load picard module load java export LANG=C

sample sheet file

set sheet = inputs/sample-sheet.tsv_

[tsirigos]

1. Can you list the contents of the pipeline/align directory (use tree so we can see the entire structure).
2. Does your scheduler submit jobs using qsub? If not, we will need to modify a few scripts to make it work for you.
3. We are preparing a tar file with the contents of the hic-bench/data directory (which includes genomes)

[quzhouxiachuan]

Hi Here is my pipeline/align directory

. ├── clean.tcsh -> code/clean.tcsh ├── code -> ../code ├── inpdirs │   └── inputs -> ../inputs ├── inputs -> ../inputs ├── params │   └── params.bowtie2.tcsh ├── results └── run

[tsirigos]

Looks normal, try the following from this directory: ./run --dry-run

Also, can you list the content of the run script. On May 25, 2016 8:58 PM, "quzhouxiachuan" notifications@github.com wrote:

Hi Here is my pipeline/align directory

. ├── clean.tcsh -> code/clean.tcsh ├── code -> ../code ├── inpdirs │ └── inputs -> ../inputs ├── inputs -> ../inputs ├── params │ └── params.bowtie2.tcsh ├── results └── run

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/NYU-BFX/hic-bench/issues/1#issuecomment-221750333

[quzhouxiachuan]

When I ran ./run --dry-run in align folder, it showed the following(I also put the r script here)

_[ydw529@quser12 align]$ ./run --dry-run === Operation = align ============= USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES [ydw529@quser12 align]$

[ydw529@quser12 align]$ vi run

% This step performs alignment of paired-end Hi-C reads.

% TABLES:

% FIGURES:

% process command-line inputs

if ($#argv > 1) then grep '^##' $0 | scripts-send2err exit endif set opt = "$1"

% setup

set op = align set inpdirs = "inputs" set results = results scripts-create-path $results/ scripts-send2err "=== Operation = $op =============" set resources = 16 set cmd = "./code/code.main/scripts-qsub-wrapper $resources ./code/hicseq-$op.tcsh"

% symlink fastq to results (required for first step of the pipeline)

if (! -e inputs/$results) then (cd inputs; ln -s fastq $results) endif

% generate run script

Rscript ./code/code.main/pipeline-master-explorer.r -v "$cmd" $results/$op "params/params.*.tcsh" "$inpdirs" "" "sample" 1

% run and wait until done!

if ("$opt" != "--dry-run") scripts-submit-jobs ./$results/.db/run_

[tsirigos]

This is weird, but it should be sth simple. Can you try this on your command line: Rscript ./code/code.main/pipeline-master-explorer.r --help

Also, here is the data you need, let me know if you can download: http://nyuepi:bl00045nyuepi@genome.med.nyu.edu:/external/aifantislab/hicbench-data-directory.tgz

[quzhouxiachuan]

This command line works fine:

[ydw529@quser12 align]$ Rscript ./code/code.main/pipeline-master-explorer.r --help Usage: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES

Options: -v, --verbose Print more messages.

-S SAMPLE-SHEET, --sample-sheet=SAMPLE-SHEET
    Sample sheet file name (required) [default "inputs/sample-sheet.tsv"].

-F FILTER-BRANCH, --filter-branch=FILTER-BRANCH
    Regular expression for filtering input branches [default ""].

--exclude-branch=EXCLUDE-BRANCH
    Regular expression for excluding input branches [default ""].

--exclude-obj=EXCLUDE-OBJ
    Regular expression for excluding input objects [default ""].

--exclude-outdir=EXCLUDE-OUTDIR
    Regular expression for excluding output directories [default ""].

-h, --help
    Show this help message and exit

[quzhouxiachuan]

I successfully downloaded the data file, thanks!

[tsirigos]

Ok, so for some reason this inputs to this script are not properly set in your system. Can you modify the run script by adding these lines before the Rscript ... command:

echo "$cmd" echo $results/$op echo "params/params.*.tcsh" echo "$inpdirs" echo "sample"

Also, update your repository. I made a simple change for enable automatic installation of the required R packages for you.

[quzhouxiachuan]

I added echo before the Rscript. I haven't update my repository yet. The result is as following:

[ydw529@quser13 align]$ ./run --dry-run === Operation = align ============= ./code/code.main/scripts-qsub-wrapper 16 ./code/hicseq-align.tcsh results/align params/params.*.tcsh inputs sample USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES [ydw529@quser13 align]$

[tsirigos]

Ok try this one thing: do you see the two quotes "" right before "sample"? Change it to " ", allowing a space between the double quotes. It may be that your version of Rscript does not recognize empty strings. If this does not work, use two single quotes instead of the space. We can discuss tomorrow on the phone in more detail.

On May 25, 2016 11:42 PM, "quzhouxiachuan" notifications@github.com wrote:

I added echo before the Rscript. I haven't update my repository yet. The result is as following:

[ydw529@quser13 align]$ ./run --dry-run

=== Operation = align ============= ./code/code.main/scripts-qsub-wrapper 16 ./code/hicseq-align.tcsh results/align params/params.*.tcsh inputs sample

USAGE: pipeline-master-explorer.r [OPTIONS] SCRIPT OUTDIR-PREFIX PARAM-SCRIPTS INPUT-BRANCHES SPLIT-VARIABLE OUTPUT-OBJECT-VARIABLE TUPLES [ydw529@quser13 align]$

— You are receiving this because you commented.

Reply to this email directly or view it on GitHub

[quzhouxiachuan]

I tried both single space and double space, it is still not working. Yes, we can discuss this on the phone tomorrow, feel free to contact 312-483-9916. Thanks for your help!

[tsirigos]

Ok, I will call you at 6.30pm EST.

[quzhouxiachuan]

Sounds good, thanks!

[tsirigos]

One more thing to try, from the pipeline/align directory: Rscript ./code/code.main/pipeline-master-explorer.r -v "./code/code.main/scripts-qsub-wrapper 16 ./code/hicseq-align.tcsh" results/align "params/params.*.tcsh" "inputs" "" "sample" 1

[tsirigos]

Could it be that you are in "Chinese mode" in your terminal? Some people here had this issue...

[quzhouxiachuan]

Hi Tsirigos,

I don't believe that is the issue here as I bought the computer in Chicago and I did not do anything particular to change the language to Chinese. But to be safe, I set the language with the following command: "sudo languagesetup" and select English as first language. I also tried this " export LANG=en_US.utf8". Then I ran the command, it still shows the same error.

[quzhouxiachuan]

I used this command to run and it worked: R --slave --vanilla --file=./code/code.main/pipeline-master-explorer.r --args -v "./code/code.main/scripts-qsub-wrapper 16 ./code/hicseq-align.tcsh" results/align "params/params.bowtie2.tcsh" "inputs" "" "sample" 1