Open volkansevim opened 1 year ago
Snakelike fails on rule download_asp because this file on AWS is missing: 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
Below is the complete log.
The flag 'directory' used in rule all is only valid for outputs, not inputs. Building DAG of jobs... Pulling singularity image docker://satijalab/azimuth-references:vitessce. Pulling singularity image docker://satijalab/azimuth-references:latest. Using shell: /usr/bin/bash Provided cores: 8 Rules claiming more threads will be scaled down. Job stats: job count min threads max threads ------------------------------ ------- ------------- ------------- all 1 1 1 download_BBI_individual_organs 14 1 1 download_BBI_metadata 1 1 1 download_BBI_subsampled 1 1 1 download_asp 1 1 1 download_enge 1 1 1 download_lind 1 1 1 export_zarr 1 1 1 make_BBI_query 1 1 1 make_BBI_reference 1 1 1 preprocess_BBI_reference 1 1 1 setup_asp 1 1 1 setup_enge 1 1 1 setup_lind 1 1 1 total 27 1 1 Select jobs to execute... [Fri Mar 17 18:53:04 2023] rule download_BBI_individual_organs: output: data/Stomach_gene_count.RDS jobid: 18 reason: Missing output files: data/Stomach_gene_count.RDS wildcards: sample=Stomach resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_BBI_individual_organs: output: data/Liver_gene_count.RDS jobid: 11 reason: Missing output files: data/Liver_gene_count.RDS wildcards: sample=Liver resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_BBI_subsampled: output: data/gene_count_sampled.RDS jobid: 3 reason: Missing output files: data/gene_count_sampled.RDS resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_BBI_individual_organs: output: data/Thymus_gene_count.RDS jobid: 19 reason: Missing output files: data/Thymus_gene_count.RDS wildcards: sample=Thymus resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_BBI_metadata: output: data/df_cell.RDS, data/df_gene.RDS jobid: 4 reason: Missing output files: data/df_gene.RDS, data/df_cell.RDS resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_lind: output: data/lind_data/counts.tsv jobid: 23 reason: Missing output files: data/lind_data/counts.tsv resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_enge: output: data/enge_data, data/enge_data/enge.tar jobid: 25 reason: Missing output files: data/enge_data/enge.tar resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:04 2023] rule download_BBI_individual_organs: output: data/Heart_gene_count.RDS jobid: 10 reason: Missing output files: data/Heart_gene_count.RDS wildcards: sample=Heart resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:06 2023] Finished job 18. 1 of 27 steps (4%) done Select jobs to execute... [Fri Mar 17 18:53:06 2023] rule download_BBI_individual_organs: output: data/Lung_gene_count.RDS jobid: 12 reason: Missing output files: data/Lung_gene_count.RDS wildcards: sample=Lung resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:08 2023] Finished job 19. 2 of 27 steps (7%) done Select jobs to execute... [Fri Mar 17 18:53:08 2023] rule download_BBI_individual_organs: output: data/Pancreas_gene_count.RDS jobid: 14 reason: Missing output files: data/Pancreas_gene_count.RDS wildcards: sample=Pancreas resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:11 2023] Finished job 4. 3 of 27 steps (11%) done Select jobs to execute... [Fri Mar 17 18:53:11 2023] rule download_BBI_individual_organs: output: data/Cerebrum_gene_count.RDS jobid: 8 reason: Missing output files: data/Cerebrum_gene_count.RDS wildcards: sample=Cerebrum resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:21 2023] Finished job 12. 4 of 27 steps (15%) done Select jobs to execute... [Fri Mar 17 18:53:21 2023] rule download_BBI_individual_organs: output: data/Adrenal_gene_count.RDS jobid: 6 reason: Missing output files: data/Adrenal_gene_count.RDS wildcards: sample=Adrenal resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:30 2023] Finished job 11. 5 of 27 steps (19%) done Select jobs to execute... [Fri Mar 17 18:53:30 2023] rule download_asp: output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv jobid: 21 reason: Missing output files: data/asp_data/md_filtered.tsv, data/asp_data/counts_filtered.tsv resources: tmpdir=/tmp Activating singularity image /prj/refs/azimuth-references/human_fetus/.snakemake/singularity/e4faa2430c925e8c50307f5d2609b777.simg [Fri Mar 17 18:53:31 2023] Error in rule download_asp: jobid: 21 output: data/asp_data/counts_filtered.tsv, data/asp_data/md_filtered.tsv shell: mkdir -p data/asp_data cd data/asp_data wget 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip' unzip mbvhhf8m62-2.zip unzip Filtered/'Developmental_heart_filtered_scRNA-seq_and_meta_data.zip' gunzip share_files/* mv share_files/all_cells_count_matrix_filtered.tsv counts_filtered.tsv mv share_files/all_cells_meta_data_filtered.tsv md_filtered.tsv echo "ASP data downloaded on: $(date)" > ../../logs/download_asp.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!) [Fri Mar 17 18:53:34 2023] Finished job 10. 6 of 27 steps (22%) done [Fri Mar 17 18:53:36 2023] Finished job 3. 7 of 27 steps (26%) done [Fri Mar 17 18:53:37 2023] Finished job 14. 8 of 27 steps (30%) done [Fri Mar 17 18:53:52 2023] Finished job 6. 9 of 27 steps (33%) done [Fri Mar 17 18:55:04 2023] Finished job 8. 10 of 27 steps (37%) done [Fri Mar 17 18:55:09 2023] Finished job 23. 11 of 27 steps (41%) done [Fri Mar 17 18:55:59 2023] Finished job 25. 12 of 27 steps (44%) done Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2023-03-17T184850.349820.snakemake.log
Snakelike fails on rule download_asp because this file on AWS is missing: 'https://md-datasets-cache-zipfiles-prod.s3.eu-west-1.amazonaws.com/mbvhhf8m62-2.zip'
Below is the complete log.