search

ENCODE-DCC / atac-seq-pipeline

ENCODE ATAC-seq pipeline
MIT License
393 stars 174 forks source link

test JSON timing out. #389

Open astephen333 opened 2 years ago

astephen333 commented 2 years ago

Describe the bug

What I ran: (base) [azstephe@compute-1-13 atac-seq-pipeline]$ caper run atac.wdl -i https://storage.googleapis.com/encode-pipeline-test-samples/encode-atac-seq-pipeline/ENCSR356KRQ_subsampled.json --conda

Printed on command line: 2022-06-23 14:30:45,481|caper.cli|INFO| Cromwell stdout: /home/azstephe/atac-seq-pipeline/cromwell.out.9 2022-06-23 14:30:45,497|caper.caper_base|INFO| Creating a timestamped temporary directory. /home/azstephe/atac-seq-pipeline/.caper_tmp/atac/20220623_143045_482418 2022-06-23 14:30:45,497|caper.caper_runner|INFO| Localizing files on work_dir. /home/azstephe/atac-seq-pipeline/.caper_tmp/atac/20220623_143045_482418 2022-06-23 14:30:45,997|autouri.autouri|INFO| cp: (ead661d1) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz, dest=/home/azstephe/atac-seq-pipeline/.caper_tmp/caf534ed3cf684406e731d19be272b4a/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz 2022-06-23 14:30:49,842|autouri.autouri|INFO| cp: (ead661d1) skipped due to name_size_match. size=872949833, mt=1549739698.0 2022-06-23 14:30:49,843|autouri.autouri|INFO| cp: (cae62311) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz, dest=/home/azstephe/atac-seq-pipeline/.caper_tmp/f43b63a83784d3ec8055f1a22168ed89/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz 2022-06-23 14:30:52,255|autouri.autouri|INFO| cp: (cae62311) skipped due to md5_match. md5=05297d96dd1f7cfb45a7b637d6dd7036 2022-06-23 14:30:52,256|autouri.autouri|INFO| cp: (3163373a) started. src=https://www.encodeproject.org/files/ENCFF356LFX/@@download/ENCFF356LFX.bed.gz, dest=/home/azstephe/atac-seq-pipeline/.caper_tmp/f183dcba5d34f959d8b55ed438ee2e22/ENCFF356LFX.bed.gz 2022-06-23 14:30:54,660|autouri.autouri|INFO| cp: (3163373a) skipped due to md5_match. md5=393688b4f06c9ce26165d47433dd8c37 2022-06-23 14:30:55,541|autouri.autouri|INFO| cp: (2a5cb466) started. src=https://www.encodeproject.org/files/GRCh38_EBV.chrom.sizes/@@download/GRCh38_EBV.chrom.sizes.tsv, dest=/home/azstephe/atac-seq-pipeline/.caper_tmp/c52f52c7bfa357f55a39b1de7e4d0b0c/GRCh38_EBV.chrom.sizes.tsv 2022-06-23 14:30:56,306|autouri.autouri|INFO| cp: (2a5cb466) skipped due to md5_match. md5=c95303fb77cc3e11d50e3c3a4b93b3fb 2022-06-23 14:30:56,308|autouri.autouri|INFO| cp: (bf1009b3) started. src=https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz, dest=/home/azstephe/atac-seq-pipeline/.caper_tmp/3ff4ac4c3f59d096b1a3842a182072ae/ENCFF110MCL.tar.gz Traceback (most recent call last): File "/home/azstephe/miniconda3/bin/caper", line 13, in main() File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/cli.py", line 710, in main return runner(parsed_args, nonblocking_server=nonblocking_server) File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/cli.py", line 252, in runner subcmd_run(c, args) File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/cli.py", line 383, in subcmd_run thread = caper_runner.run( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/caper_runner.py", line 409, in run maybe_remote_file = self.localize_on_backend_if_modified( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/caper_base.py", line 162, in localize_on_backend_if_modified f_loc = self.localize_on_backend( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/caper/caper_base.py", line 147, in localize_on_backend return AutoURI(f).localize_on( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 421, in localize_on return AutoURI.localize( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 643, in localize maybe_modified_contents, modified = fnc_recurse( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/loc_aux.py", line 56, in recurse_json modified = recurse_dict(d, fnc) File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/loc_aux.py", line 34, in recurse_dict modified |= recurse_dict( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/loc_aux.py", line 42, in recurse_dict newval, modified = fnc(d) File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 628, in fnc_loc return cls.localize( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 643, in localize maybe_modified_contents, modified = fnc_recurse( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/loc_aux.py", line 83, in recurse_tsv newval, modified = fnc(v) File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 628, in fnc_loc return cls.localize( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 660, in localize src_uri.cp( File "/home/azstephe/miniconda3/lib/python3.9/site-packages/autouri/autouri.py", line 298, in cp with d.get_lock(no_lock=no_lock): File "/home/azstephe/miniconda3/lib/python3.9/site-packages/filelock/_api.py", line 220, in enter self.acquire() File "/home/azstephe/miniconda3/lib/python3.9/site-packages/filelock/_api.py", line 183, in acquire raise Timeout(self._lock_file) filelock._error.Timeout: The file lock '/home/azstephe/atac-seq-pipeline/.caper_tmp/3ff4ac4c3f59d096b1a3842a182072ae/ENCFF110MCL.tar.gz.lock' could not be acquired.

OS/Platform

Caper configuration file

backend=slurm

SLURM partition. DEFINE ONLY IF REQUIRED BY YOUR CLUSTER'S POLICY.

You must define it for Stanford Sherlock.

slurm-partition=pfen2

SLURM account. DEFINE ONLY IF REQUIRED BY YOUR CLUSTER'S POLICY.

You must define it for Stanford SCG.

Local directory for localized files and Cromwell's intermediate files.

If not defined then Caper will make .caper_tmp/ on CWD or local-out-dir.

/tmp is not recommended since Caper store localized data files here.

local-loc-dir=

This parameter defines resource parameters for Caper's leader job only.

slurm-leader-job-resource-param=-t 48:00:00 --mem 4G

This parameter defines resource parameters for submitting WDL task to job engine.

It is for HPC backends only (slurm, sge, pbs and lsf).

It is not recommended to change it unless your cluster has custom resource settings.

See https://github.com/ENCODE-DCC/caper/blob/master/docs/resource_param.md for details.

slurm-resource-param=-n 1 --ntasks-per-node=1 --cpus-per-task=${cpu} ${if defined(memory_mb) then "--mem=" else ""}${memory_mb}${if defined(memory_mb) then "M" else ""} ${if defined(time) then "--time=" else ""}${time*60} ${if defined(gpu) then "--gres=gpu:" else ""}${gpu}

cromwell=/home/azstephe/.caper/cromwell_jar/cromwell-65.jar womtool=/home/azstephe/.caper/womtool_jar/womtool-65.jar

Input JSON file

test JSON.

leepc12 commented 2 years ago

I think ENCODE server was temporarily unavailable for hosting https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz

Please check if you can download it manually

$ wget https://www.encodeproject.org/files/ENCFF110MCL/@@download/ENCFF110MCL.tar.gz

If it works, re-run the Caper command line.

hyjforesight commented 2 years ago

hello @astephen333 @leepc12 I met the same issue as @astephen333 .

filelock._error.Timeout: The file lock '/home/hyjforesight/.caper_tmp/72041e824adb499a4974d85ba31fe30d/mm10_no_alt_analysis_set_ENCODE.fasta.gz.lock' could not be acquired.

I can download the file manually, but the pipeline still doesnt work.

wget https://www.encodeproject.org/files/mm10_no_alt_analysis_set_ENCODE/@@download/mm10_no_alt_analysis_set_ENCODE.fasta.gz

Could you please help me with this issue? Thanks! Best, YJ

Rafaelsoler13 commented 1 year ago

For me, the same problem was happening. It was due to slow download speed from the ENCODE server. What I did to solve it, is to download the file with a VPN, and put the file inside the corresponding folder. After this, the pipeline recognizes the file by size or md:

2023-04-26 18:29:13,467|autouri.autouri|INFO| cp: (87cc8c89) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz, dest=~/ENCODE_workflow/temp_files_caper/caf534ed3cf684406e731d19be272b4a/GRCh38_no_alt_analysis_set_GCA_000001405.15.fasta.gz
2023-04-26 18:29:16,305|autouri.autouri|INFO| cp: (87cc8c89) skipped due to name_size_match. size=872949833, mt=1549739698.0
2023-04-26 18:29:16,305|autouri.autouri|INFO| cp: (800336c2) started. src=https://www.encodeproject.org/files/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only/@@download/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz, dest=~/ENCODE_workflow/temp_files_caper/f43b63a83784d3ec8055f1a22168ed89/GRCh38_no_alt_analysis_set_GCA_000001405.15_mito_only.fasta.gz
2023-04-26 18:29:17,872|autouri.autouri|INFO| cp: (800336c2) skipped due to md5_match. md5=05297d96dd1f7cfb45a7b637d6dd7036

Best,

Rafael