Fatal error: atac_dnase_pipelines/modules/sys.bds, line 143, pos 2. Exec failed.

[yeouyang]

Hi, I am trying to run ATAC-seq pipeline. At the step of bds atac.bds... error like that:

Fatal error: /root/atac_dnasepipelines/modules/sys.bds, line 143, pos 2. Exec failed. Exit value : 1 Command : hostname -f 1> /root/data/12/cmd-4215330719029919960 git.bds, line 4 : include "sys.bds" sys.bds, line 4 : include "string.bds" sys.bds, line 21 : init_base() sys.bds, line 24 : void init_base() { sys.bds, line 29 : hostname = get_hostname() sys.bds, line 175 : string get_hostname() { sys.bds, line 176 : return get_stdout("hostname -f").replace("\n","") sys.bds, line 140 : string getstdout( string cmd ) { sys.bds, line 143 : sys $cmd 1> $cmd

[leepc12]

Please post a full error log. Did you run pipeline in a docker container?

Jin

On Sun, Nov 5, 2017 at 10:49 PM, yeouyang notifications@github.com wrote:

Hi, I am trying to run ATAC-seq pipeline. At the step of bds atac.bds... error like that:

Fatal error: /root/atac_dnasepipelines/modules/sys.bds, line 143, pos 2. Exec failed. Exit value : 1 Command : hostname -f 1> /root/data/12/cmd-4215330719029919960 git.bds, line 4 : include "sys.bds" sys.bds, line 4 : include "string.bds" sys.bds, line 21 : init_base() sys.bds, line 24 : void init_base() { sys.bds, line 29 : hostname = get_hostname() sys.bds, line 175 : string get_hostname() { sys.bds, line 176 : return get_stdout("hostname -f").replace("\n","") sys.bds, line 140 : string getstdout( string cmd ) { sys.bds, line 143 : sys $cmd 1> $cmd

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_I-IWvMPoroQ9Y-23bWB6ZE3orpHks5szqwXgaJpZM4QSzPb .

[yeouyang]

hostname: Name or service not known Fatal error: /root/atac_dnasepipelines/modules/sys.bds, line 143, pos 2. Exec failed. Exit value : 1 Command : hostname -f 1> /root/data/12/cmd-4215330719029919960 git.bds, line 4 : include "sys.bds" sys.bds, line 4 : include "string.bds" sys.bds, line 21 : init_base() sys.bds, line 24 : void init_base() { sys.bds, line 29 : hostname = get_hostname() sys.bds, line 175 : string get_hostname() { sys.bds, line 176 : return get_stdout("hostname -f").replace("\n","") sys.bds, line 140 : string getstdout( string cmd ) { sys.bds, line 143 : sys $cmd 1> $cmd

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

Not in container, and all things is here.

[leepc12]

Can you try the following command lines for debugging info?

$ hostname

$ hostname -f

$ cat /etc/hosts

Jin

On Sun, Nov 5, 2017 at 11:34 PM, yeouyang notifications@github.com wrote:

hostname: Name or service not known Fatal error: /root/atac_dnasepipelines/modules/sys.bds, line 143, pos 2. Exec failed. Exit value : 1 Command : hostname -f 1> /root/data/12/cmd-4215330719029919960 git.bds, line 4 : include "sys.bds" sys.bds, line 4 : include "string.bds" sys.bds, line 21 : init_base() sys.bds, line 24 : void init_base() { sys.bds, line 29 : hostname = get_hostname() sys.bds, line 175 : string get_hostname() { sys.bds, line 176 : return get_stdout("hostname -f").replace("\n","") sys.bds, line 140 : string getstdout( string cmd ) { sys.bds, line 143 : sys $cmd 1> $cmd

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

Not in container, and all things is here.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342067915, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_Hwj0cvpJW4dbtKLCyx58rsXkoTUks5szraFgaJpZM4QSzPb .

[leepc12]

Can you modify modules/sys.bds and try again?

Replace

string get_hostname() { return get_stdout("hostname -f").replace("\n","") }

with

string get_hostname() { out := get_stdout("hostname -f || true").replace("\n","") if (out.startsWith("hostname: ")) return "unknown" else return out }

Please let me know if this works so that I can make a hot fix for it.

Thanks,

Jin

On Sun, Nov 5, 2017 at 11:42 PM, Jin leepc12@gmail.com wrote:

Can you try the following command lines for debugging info?

$ hostname

$ hostname -f

$ cat /etc/hosts

Jin

On Sun, Nov 5, 2017 at 11:34 PM, yeouyang notifications@github.com wrote:

hostname: Name or service not known Fatal error: /root/atac_dnase_pipelines/modules/sys.bds, line 143, pos

2. Exec failed. Exit value : 1 Command : hostname -f 1> /root/data/12/cmd_-4215330719029919960 git.bds, line 4 : include "sys.bds" sys.bds, line 4 : include "string.bds" sys.bds, line 21 : init_base() sys.bds, line 24 : void init_base() { sys.bds, line 29 : hostname = get_hostname() sys.bds, line 175 : string get_hostname() { sys.bds, line 176 : return get_stdout("hostname -f").replace("\n","") sys.bds, line 140 : string getstdout( string cmd ) { sys.bds, line 143 : sys $cmd 1> $cmd

Creating checkpoint file: Config or command line option disabled checkpoint file creation, nothing done.

Not in container, and all things is here.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342067915, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_Hwj0cvpJW4dbtKLCyx58rsXkoTUks5szraFgaJpZM4QSzPb .

[yeouyang]

Sorry for late. I tried your advice, and error like this:

hostname: Name or service not known

== git info Latest git commit : 05a35648a5883f4383ad83db9d2c5c126aaf7aac (Mon Nov 6 07:50:02 2017)

== configuration file info Hostname : Configuration file : Environment file : /root/atac_dnase_pipelines/default.env

== parallelization info No parallel jobs : false Maximum # threads : 8

== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false Java temporary directory : ${TMPDIR}

Info: Environments module not found on your system (e.g. /etc/profile.d/modules.sh). Ignoring shell env. parameters like '-mod'.

== shell environment info Conda env. : Conda env. for python3 : Conda bin. directory :

Shell cmd. for init. : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for init.(py3) : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Cluster task min. len. : 60

Cluster task delay : 0

== output directory/title info Output dir. : /root/data/12/out Title (prefix) : 12

== species settings Species : Species file :

Species name (WashU browser) : Ref. genome seq. fasta : Chr. sizes file : Black list bed : Ref. genome seq. dir. :

== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :

== report settings URL root for output directory : Genome coord. for browser tracks :

== align multimapping settings

alignments reported for multimapping : 0

== align bowtie2 settings Bowtie2 index : Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :

== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G

== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false

== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G

== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :

== callpeak macs2 settings Genome size (hs,mm) : Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :

== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false

== callpeak etc settings

of top peaks to pick up in peak files : 500000

== IDR settings Append IDR threshold to IDR out_dir : false

== ATAQC settings TSS enrichment bed : DNase bed for ataqc : Promoter bed for ataqc : Enhancer bed for ataqc : Reg2map for ataqc : Reg2map_bed for ataqc : Roadmap metadata for ataqc : Max. memory for ATAQC : 20G Walltime for ATAQC : 47h

== atac pipeline settings Type of pipeline : atac-seq Align only : false

reads to subsample replicates (0 if no subsampling) : 0

reads to subsample for cross-corr. analysis : 25000000

No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : false Automatically find/trim adapters : false

== checking atac parameters ... 00:00:00.249 Error (/root/atac_dnase_pipelines/modules/align_bowtie2.bds, line 41, pos 3): Bowtie2 index (-bwt2_idx) doesn't exists! (file: .1.bt2 or .1.bt2l)

[leepc12]

Can you try the following commands for more debugging info?

$ cat /root/atac_dnase_pipelines/default.env

Did you install genome data ( https://github.com/kundajelab/atac_dnase_pipelines#genome-data) and specify species in the command line (e.g. -species hg38)?

Jin

On Mon, Nov 6, 2017 at 12:28 AM, yeouyang notifications@github.com wrote:

Sorry for late. I tried your advice, and error like this:

hostname: Name or service not known

== git info Latest git commit : 05a3564 https://github.com/kundajelab/atac_dnase_pipelines/commit/05a35648a5883f4383ad83db9d2c5c126aaf7aac (Mon Nov 6 07:50:02 2017)

== configuration file info Hostname : Configuration file : Environment file : /root/atac_dnase_pipelines/default.env

== parallelization info No parallel jobs : false Maximum # threads : 8

== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false Java temporary directory : ${TMPDIR}

Info: Environments module not found on your system (e.g. /etc/profile.d/modules.sh). Ignoring shell env. parameters like '-mod'.

== shell environment info Conda env. : Conda env. for python3 : Conda bin. directory :

Shell cmd. for init. : export PATH=/root/atacdnase pipelines/.:/root/atac_dnasepipelines/modules:/root/atac dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for init.(py3) : export PATH=/root/atacdnase pipelines/.:/root/atac_dnasepipelines/modules:/root/atac dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Cluster task min. len. : 60

Cluster task delay : 0

== output directory/title info Output dir. : /root/data/12/out Title (prefix) : 12

== species settings Species : Species file :

Species name (WashU browser) : Ref. genome seq. fasta : Chr. sizes file : Black list bed : Ref. genome seq. dir. :

== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :

== report settings URL root for output directory : Genome coord. for browser tracks :

== align multimapping settings alignments reported for multimapping : 0

== align bowtie2 settings Bowtie2 index : Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :

== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G

== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false

== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G

== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :

== callpeak macs2 settings Genome size (hs,mm) : Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :

== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false

== callpeak etc settings of top peaks to pick up in peak files : 500000

== IDR settings Append IDR threshold to IDR out_dir : false

== ATAQC settings TSS enrichment bed : DNase bed for ataqc : Promoter bed for ataqc : Enhancer bed for ataqc : Reg2map for ataqc : Reg2map_bed for ataqc : Roadmap metadata for ataqc : Max. memory for ATAQC : 20G Walltime for ATAQC : 47h

== atac pipeline settings Type of pipeline : atac-seq Align only : false reads to subsample replicates (0 if no subsampling) : 0 reads to subsample for cross-corr. analysis : 25000000

No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : false Automatically find/trim adapters : false

== checking atac parameters ... 00:00:00.249 Error (/root/atac_dnase_pipelines/modules/align_bowtie2.bds, line 41, pos 3): Bowtie2 index (-bwt2_idx) doesn't exists! (file: .1.bt2 or .1.bt2l)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342078188, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_ODwUVrevKNev87GUfHkGk4wmeH_ks5szsNFgaJpZM4QSzPb .

[yeouyang]

I have install hg19 and built index.

Get hostname with the following command:

$ hostname -f

Configure environment per hostname:

[hostname1]

...

Use the same environment for multiple hostnames:

[hostname2, hostname3, ...]

...

Using group

[hostname1, hostname2, ... : group]

[group]

...

Using an asterisk in hostnames (IMPORTANT: only one * is allowed in hostnames)

[host*name1]

[hostname2, hostname3]

Stanford Kundaje group clusters (out of SGE)

[vayu, mitra, durga] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /software/miniconda3/bin species_file = $script_dir/species/kundaje.conf unlimited_mem_wt= true # unlimited max. memory and walltime on Kundaje clusters nice = 20 nth = 3

Stanford Kundaje group clusters (controlled with SGE)

[nandi, kali, amold, wotan, kadru, surya, indra] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /software/miniconda3/bin species_file = $script_dir/species/kundaje.conf unlimited_mem_wt= true # unlimited max. memory and walltime on Kundaje clusters system = sge nice = 20 nth = 3

Stanford SCG4

[scg.stanford.edu, scg.local, carmack.stanford.edu, crick.stanford.edu] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /ifs/scratch/leepc12/sw/miniconda3/bin species_file = $script_dir/species/scg.conf nth = 4 # number of threads for each pipeline system = sge # force to use SGE (Sun Grid Engine) on SCG3/4 even though a user doesn't explicitly specify SGE on command line with 'bds -s sge atac.bds ...' cluster_task_delay = 10 # for NFS delayed write

Stanford Sherlock clusters

[sherlock.stanford.edu, sh-.local, sh-.int, sh-ln.stanford.edu] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /scratch/users/leepc12/bds_pipeline_software/miniconda3/bin species_file = $script_dir/species/sherlock.conf nth = 4 # number of threads for each pipeline system = slurm cluster_task_delay = 30 # for NFS delayed write

default

[default] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = species_file = /root/data/bds_pipeline_genome_data/bds_atac_species.conf # DEF_SPECIES_FILE: DO NOT MODIFY THIS COMMENT (install_genome_data.sh WILL NOT WORK).

[leepc12]

What was your command line to run the pipeline?

On Mon, Nov 6, 2017 at 12:41 AM, yeouyang notifications@github.com wrote:

I have install hg19 and built index. Get hostname with the following command: $ hostname -f Configure environment per hostname: [hostname1] ... Use the same environment for multiple hostnames: [hostname2, hostname3, ...] ... Using group [hostname1, hostname2, ... : group] [group] ... Using an asterisk in hostnames (IMPORTANT: only one is allowed in hostnames) [hostname1] [hostname2, hostname3] Stanford Kundaje group clusters (out of SGE)

[vayu, mitra, durga] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /software/miniconda3/bin species_file = $script_dir/species/kundaje.conf unlimited_mem_wt= true # unlimited max. memory and walltime on Kundaje clusters nice = 20 nth = 3 Stanford Kundaje group clusters (controlled with SGE)

[nandi, kali, amold, wotan, kadru, surya, indra] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /software/miniconda3/bin species_file = $script_dir/species/kundaje.conf unlimited_mem_wt= true # unlimited max. memory and walltime on Kundaje clusters system = sge nice = 20 nth = 3 Stanford SCG4

[scg.stanford.edu, scg.local, carmack.stanford.edu, crick.stanford.edu] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /ifs/scratch/leepc12/sw/miniconda3/bin species_file = $script_dir/species/scg.conf nth = 4 # number of threads for each pipeline system = sge # force to use SGE (Sun Grid Engine) on SCG3/4 even though a user doesn't explicitly specify SGE on command line with 'bds -s sge atac.bds ...' cluster_task_delay = 10 # for NFS delayed write Stanford Sherlock clusters

[sherlock.stanford.edu, sh-.local, sh-.int, sh-ln.stanford.edu] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = /scratch/users/leepc12/bds_pipeline_software/miniconda3/ bin species_file = $script_dir/species/sherlock.conf nth = 4 # number of threads for each pipeline system = slurm cluster_task_delay = 30 # for NFS delayed write default

[default] conda_env = bds_atac conda_env_py3 = bds_atac_py3 conda_bin_dir = species_file = /root/data/bds_pipeline_genome_data/bds_atac_species.conf

DEF_SPECIES_FILE: DO NOT MODIFY THIS COMMENT (install_genome_data.sh WILL

NOT WORK).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342080824, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_NCPyiLkr92k_6FWzCOYxVrDy5RXks5szsZBgaJpZM4QSzPb .

[yeouyang]

bds /root/atac_dnase_pipelines/atac.bds -species hg19 -4 -fastq1_1 R1.fastq.gz -fastq1_2 R2.fastq.gz

[leepc12]

Can you remove -4 and try again? What is -4 for?

On Mon, Nov 6, 2017 at 12:45 AM, yeouyang notifications@github.com wrote:

bds /root/atac_dnase_pipelines/atac.bds -species hg19 -4 -fastq1_1 R1.fastq.gz -fastq1_2 R2.fastq.gz

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342081596, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_CtM0EemVIrOwx5npR3jPpMVma0xks5szscOgaJpZM4QSzPb .

[yeouyang]

Sorry, I remove -4, and still the same error. 4 is regarded as threads.

[leepc12]

Yes, you should use -nth 4 instead of -4.

Please replace the following in modules/sys.bds

string getstdout( string cmd ) { rnd := randInt() cmd := "cmd$rnd".path() sys $cmd 1> $cmd ret := cmd.read() sys rm -f $cmd return rm_str_at_end(ret,"\n") }

with

string getstdout( string cmd ) { rnd := randInt() cmd := "cmd$rnd".path() sys $cmd 1> $cmd || true ret := cmd.read() sys rm -f $cmd return rm_str_at_end(ret,"\n") }

And then try again. Please post a new full log here even if you get the same error.

Jin

On Mon, Nov 6, 2017 at 12:51 AM, yeouyang notifications@github.com wrote:

Sorry, I remove -4, and still the same error. 4 is regarded as threads.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342083103, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_HdxnBJsKbCPU7y0GrBAmYyc7oujks5szsiVgaJpZM4QSzPb .

[yeouyang]

I do the replacement, and error like that: `hostname: Name or service not known

== git info Latest git commit : 05a35648a5883f4383ad83db9d2c5c126aaf7aac (Mon Nov 6 07:50:02 2017)

== configuration file info Hostname : Configuration file : Environment file : /root/atac_dnase_pipelines/default.env

== parallelization info No parallel jobs : false Maximum # threads : 4

== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false Java temporary directory : ${TMPDIR}

Info: Environments module not found on your system (e.g. /etc/profile.d/modules.sh). Ignoring shell env. parameters like '-mod'.

== shell environment info Conda env. : Conda env. for python3 : Conda bin. directory :

Shell cmd. for init. : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for init.(py3) : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Cluster task min. len. : 60

Cluster task delay : 0

== output directory/title info Output dir. : /root/data/12/out Title (prefix) : 12 Reading parameters from section (default) in file(/root/atac_dnase_pipelines/default.env)...

== species settings Species : hg19 Species file :

Species name (WashU browser) : hg19 Ref. genome seq. fasta : Chr. sizes file : Black list bed : Ref. genome seq. dir. :

== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :

== report settings URL root for output directory : Genome coord. for browser tracks :

== align multimapping settings

alignments reported for multimapping : 0

== align bowtie2 settings Bowtie2 index : Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :

== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G

== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false

== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G

== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :

== callpeak macs2 settings Genome size (hs,mm) : Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :

== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false

== callpeak etc settings

of top peaks to pick up in peak files : 500000

== IDR settings Append IDR threshold to IDR out_dir : false

== ATAQC settings TSS enrichment bed : DNase bed for ataqc : Promoter bed for ataqc : Enhancer bed for ataqc : Reg2map for ataqc : Reg2map_bed for ataqc : Roadmap metadata for ataqc : Max. memory for ATAQC : 20G Walltime for ATAQC : 47h

== atac pipeline settings Type of pipeline : atac-seq Align only : false

reads to subsample replicates (0 if no subsampling) : 0

reads to subsample for cross-corr. analysis : 25000000

No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : false Automatically find/trim adapters : false

== checking atac parameters ... 00:00:00.211 Error (/root/atac_dnase_pipelines/modules/align_bowtie2.bds, line 41, pos 3): Bowtie2 index (-bwt2_idx) doesn't exists! (file: .1.bt2 or .1.bt2l)`

[leepc12]

Sorry, please replace the following too.

Replace

string get_hostname() { out := get_stdout("hostname -f || true").replace("\n","") if (out.startsWith("hostname: ")) return "unknown" else return out }

with

string get_hostname() { out := get_stdout("hostname -f").replace("\n","") if (out.startsWith("hostname: ")) return "unknown" else return out }

Jin

On Mon, Nov 6, 2017 at 1:04 AM, Jin leepc12@gmail.com wrote:

Yes, you should use -nth 4 instead of -4.

Please replace the following in modules/sys.bds

string getstdout( string cmd ) { rnd := randInt() cmd := "cmd$rnd".path() sys $cmd 1> $cmd ret := cmd.read() sys rm -f $cmd return rm_str_at_end(ret,"\n") }

with

string getstdout( string cmd ) { rnd := randInt() cmd := "cmd$rnd".path() sys $cmd 1> $cmd || true ret := cmd.read() sys rm -f $cmd return rm_str_at_end(ret,"\n") }

And then try again. Please post a new full log here even if you get the same error.

Jin

On Mon, Nov 6, 2017 at 12:51 AM, yeouyang notifications@github.com wrote:

Sorry, I remove -4, and still the same error. 4 is regarded as threads.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kundajelab/atac_dnase_pipelines/issues/80#issuecomment-342083103, or mute the thread https://github.com/notifications/unsubscribe-auth/AIOd_HdxnBJsKbCPU7y0GrBAmYyc7oujks5szsiVgaJpZM4QSzPb .

[yeouyang]

hostname: Name or service not known

== git info Latest git commit : 05a35648a5883f4383ad83db9d2c5c126aaf7aac (Mon Nov 6 07:50:02 2017)

== configuration file info Hostname : Configuration file : Environment file : /root/atac_dnase_pipelines/default.env

== parallelization info No parallel jobs : false Maximum # threads : 4

== cluster/system info Walltime (general) : 5h50m Max. memory (general) : 7G Force to use a system : local Process priority (niceness) : 0 Retiral for failed tasks : 0 Submit tasks to a cluster queue : Unlimited cluster mem./walltime : false Java temporary directory : ${TMPDIR}

Info: Environments module not found on your system (e.g. /etc/profile.d/modules.sh). Ignoring shell env. parameters like '-mod'.

== shell environment info Conda env. : Conda env. for python3 : Conda bin. directory :

Shell cmd. for init. : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for init.(py3) : export PATH=/root/atac_dnase_pipelines/.:/root/atac_dnase_pipelines/modules:/root/atac_dnase_pipelines/utils:${PATH}:/bin:/usr/bin:/usr/local/bin:${HOME}/.bds; set -o pipefail; STARTTIME=$(date +%s)

Shell cmd. for fin. : TASKTIME=$[$(date +%s)-${STARTTIME}]; echo "Task has finished (${TASKTIME} seconds)."; sleep 0

Cluster task min. len. : 60

Cluster task delay : 0

== output directory/title info Output dir. : /root/data/12/out Title (prefix) : 12 Reading parameters from section (default) in file(/root/atac_dnase_pipelines/default.env)...

== species settings Species : hg19 Species file :

Species name (WashU browser) : hg19 Ref. genome seq. fasta : Chr. sizes file : Black list bed : Ref. genome seq. dir. :

== ENCODE accession settings ENCODE experiment accession : ENCODE award RFA : ENCODE assay category : ENCODE assay title : ENCODE award : ENCODE lab : ENCODE assembly genome : ENCODE alias prefix : KLAB_PIPELINE ENCODE alias suffix :

== report settings URL root for output directory : Genome coord. for browser tracks :

== align multimapping settings

alignments reported for multimapping : 0

== align bowtie2 settings Bowtie2 index : Replacement --score-min for bowtie2 : Walltime (bowtie2) : 47h Max. memory (bowtie2) : 12G Extra param. (bowtie2) :

== adapter trimmer settings Maximum allowed error rate for cutadapt : 0.10 Minimum trim. length for cutadapt -m : 5 Walltime (adapter trimming) : 23h Max. memory (adapter trimming) : 12G

== postalign bam settings MAPQ reads rm thresh. : 30 Rm. tag reads with str. : No dupe removal in filtering raw bam : false Walltime (bam filter) : 23h Max. memory (bam filter) : 12G Dup marker : picard Use sambamba markdup (instead of picard) : false

== postalign bed/tagalign settings Max. memory for UNIX shuf : 12G

== postalign cross-corr. analysis settings Max. memory for UNIX shuf : 12G User-defined cross-corr. peak strandshift : -1 Extra parameters for cross-corr. analysis :

== callpeak macs2 settings Genome size (hs,mm) : Walltime (macs2) : 23h Max. memory (macs2) : 15G Extra parameters for macs2 callpeak :

== callpeak naiver overlap settings Bedtools intersect -nonamecheck : false

== callpeak etc settings

of top peaks to pick up in peak files : 500000

== IDR settings Append IDR threshold to IDR out_dir : false

== ATAQC settings TSS enrichment bed : DNase bed for ataqc : Promoter bed for ataqc : Enhancer bed for ataqc : Reg2map for ataqc : Reg2map_bed for ataqc : Roadmap metadata for ataqc : Max. memory for ATAQC : 20G Walltime for ATAQC : 47h

== atac pipeline settings Type of pipeline : atac-seq Align only : false

reads to subsample replicates (0 if no subsampling) : 0

reads to subsample for cross-corr. analysis : 25000000

No pseudo replicates : false No ATAQC (advanced QC report) : false No Cross-corr. analysis : false Use CSEM for alignment : false Smoothing window for MACS2 : 150 DNase Seq : false IDR threshold : 0.1 Force to use ENCODE3 parameter set : false Force to use ENCODE parameter set : false Disable genome browser tracks : false p-val thresh. for overlapped peaks : 0.01 MACS2 p-val thresh. for peaks : 0.01 MACS2 p-val thresh. for BIGWIGs : 0.01 Enable IDR on called peaks : false Automatically find/trim adapters : false

== checking atac parameters ... 00:00:00.191 Error (/root/atac_dnase_pipelines/modules/align_bowtie2.bds, line 41, pos 3): Bowtie2 index (-bwt2_idx) doesn't exists! (file: .1.bt2 or .1.bt2l)

[leepc12]

Okay, I figure it out. Please unzip the attached file and replace your modules/sys.bds with it. sys.zip

Also, please run the following for more debugging info: $ cat /root/data/bds_pipeline_genome_data/bds_atac_species.conf

Jin

[yeouyang]

It seems work, up to now it finished trimming adapter. Thank you, Jin!

[leepc12]

Sounds good.