search

a-h-b / binny

GNU General Public License v3.0
27 stars 6 forks source link

Batch job submission failed: Requested node configuration is not available #46

Closed Nathhub closed 1 year ago

Nathhub commented 1 year ago

Hi there, I am struggling to identify what is the problem with the configuration of binny. I hope you can help.

The steps seem to be going well until step 9 (see log report below):

The command I use is : ./binny -c -r -t 64 config/config.default.yaml

I am using an HPC cluster having 64 CPUs per nodes and 4G per CPU. The HPC only allows me to send jobs of max 24 hours.

Below you will find the content of some config files.

Thanks in advance for your help.

Cheers, Nathan

---------------------------------------------------------------------- snakemake log -------------------

Config file /home/ndh1n17/.conda/envs/mamba/envs/binny/binny/config/config.default.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cluster nodes: 64 Job stats: job count min threads max threads

ALL 1 1 1 annotate 1 24 24 binny 1 64 64 call_contig_depth 3 1 1 format_assembly 1 1 1 mantis_checkm_marker_sets 1 64 64 merge_contig_depths 1 1 1 prepare_input_data 1 1 1 total 10 1 64

Select jobs to execute...

[Mon Feb 20 17:30:58 2023] Job 5: Preparing input. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.fa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000001_sorted.bam, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000002_sorted.bam, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000003_sorted.bam

Config file /home/ndh1n17/.conda/envs/mamba/envs/binny/binny/config/config.default.yaml is extended by additional config specified via the command line. Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 1 (use --cores to define parallelism) Rules claiming more threads will be scaled down. Provided resources: mem_mb=197589, disk_mb=197589 Select jobs to execute... [Mon Feb 20 17:33:54 2023] Finished job 5. 1 of 10 steps (10%) done Select jobs to execute...

[Mon Feb 20 17:33:54 2023] Job 4: Preparing assembly. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.fa

Submitted job 4 with external jobid 'Submitted batch job 229278'. [Mon Feb 20 17:34:08 2023] Finished job 4. 2 of 10 steps (20%) done Select jobs to execute...

[Mon Feb 20 17:34:08 2023] Job 8: annotate: Running prokkaP. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/prokka.faa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/annotation.filt.gff; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa

Submitted job 8 with external jobid 'Submitted batch job 229279'.

[Mon Feb 20 17:34:08 2023] Job 3: /dssgfs01/scratch/ndh1n17/08_bins/aa/call_contig_depth: Getting data on assembly coverage with mg reads. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_fsw.txt; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000001_sorted.bam

Submitted job 3 with external jobid 'Submitted batch job 229280'. [Mon Feb 20 17:34:08 2023] Job 8: annotate: Running prokkaP. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/prokka.faa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/annotation.filt.gff; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa

Submitted job 8 with external jobid 'Submitted batch job 229279'.

[Mon Feb 20 17:34:08 2023] Job 3: /dssgfs01/scratch/ndh1n17/08_bins/aa/call_contig_depth: Getting data on assembly coverage with mg reads. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_fsw.txt; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000001_sorted.bam

Submitted job 3 with external jobid 'Submitted batch job 229280'.

[Mon Feb 20 17:34:08 2023] Job 6: /dssgfs01/scratch/ndh1n17/08_bins/aa/call_contig_depth: Getting data on assembly coverage with mg reads. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_go.txt; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000002_sorted.bam

Submitted job 6 with external jobid 'Submitted batch job 229281'.

[Mon Feb 20 17:34:08 2023] Job 7: /dssgfs01/scratch/ndh1n17/08_bins/aa/call_contig_depth: Getting data on assembly coverage with mg reads. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_ra.txt; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.formatted.fa, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/reads_sample_000003_sorted.bam

Submitted job 7 with external jobid 'Submitted batch job 229282'. [Mon Feb 20 17:58:40 2023] Finished job 7. 3 of 10 steps (30%) done [Mon Feb 20 18:15:01 2023] Finished job 3. 4 of 10 steps (40%) done [Mon Feb 20 18:24:21 2023] Finished job 6. 5 of 10 steps (50%) done Select jobs to execute...

[Mon Feb 20 18:24:21 2023] Job 2: Merging depth files. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly.contig_depth.txt; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_fsw.txt, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_ra.txt, /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/assembly_contig_depth_aa_go.txt

Submitted job 2 with external jobid 'Submitted batch job 229285'. [Mon Feb 20 18:24:51 2023] Finished job 2. 6 of 10 steps (60%) done [Mon Feb 20 19:40:26 2023] Finished job 8. 7 of 10 steps (70%) done Select jobs to execute...

[Mon Feb 20 19:40:26 2023] Job 9: MANTIS: Running MANTIS with CheckM marker sets. Reason: Missing output files: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/mantis_out/consensus_annotation.tsv; Input files updated by another job: /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/prokka.faa

sbatch: error: Batch job submission failed: Requested node configuration is not available Error submitting jobscript (exit code 1):

Exiting because a job execution failed. Look above for error message Complete log: ../../.conda/envs/mamba/envs/binny/binny/.snakemake/log/2023-02-20T173045.483493.snakemake.log

                                                                                         VARIABLE_CONFIG                                                                                                             SNAKEMAKE_VIA_CONDA     true

SNAKEMAKE_EXTRA_ARGUMENTS LOADING_MODULES SUBMIT_COMMAND sbatch --time=24:00:00 --mem-per-cpu=4G --output=slurm-%j.out BIND_JOBS_TO_MAIN false NODENAME_VAR SLURMD_NODENAME SCHEDULER slurm_simple MAX_THREADS 64

                                                                                      config/slurm_simple.config.yaml                                                                                                     __default__:

call: "sbatch " nodes: "" partition: "" qos: "" mem_per_cpu: "--mem-per-cpu " runtime: "-t " threads: "-c " stdout: "-o binny.{rule}.{wildcards}.stdout"

                                                                                           slurm-229277.out

bash: cannot set terminal process group (-1): Inappropriate ioctl for device bash: no job control in this shell Config file /home/ndh1n17/.conda/envs/mamba/envs/binny/binny/config/config.default.yaml is extended by additional config specified via the command line. Building DAG of jobs... Creating report... Missing metadata for file /dssgfs01/scratch/ndh1n17/08_bins/aa/binny.done. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/mantis_out/output_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/mantis_out/integrated_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/mantis_out/consensus_annotation.tsv. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Missing metadata for file /dssgfs01/scratch/ndh1n17/08_bins/aa/intermediary/mantis_out. Maybe metadata was deleted or it was created using an older version of Snakemake. This is a non critical warning. Downloading resources and rendering HTML. Report created: report.html. slurm-229277.out (END)

                                                                                     config/config.default.yaml

Mask potentially disruptive contig regions (e.g. rRNA and CRISPR elements) from k-mer counting

mask_disruptive_sequences: 'True'

Extract single contig MAGs of at least 90% purity and 92.5% completeness

extract_scmags: 'True'

Will use coassembly mode, starting with contigs >= 500 bp instead of high threshold, decreasing, if set to 'on' or

if 'auto' and multiple depth files are detected

Choose between: 'auto', 'on' , 'off''

coassembly_mode: 'auto'

Binny prefilters assemblies based on N value to try and take as much information as possible into account,

while minimizing the amount of noise. Be aware that, depending on the assembly quality, low values as the N

might results in leaving out a large proportion of the assembly (if the max_cont_length cutoffs are set high as well).

NX_value: 90

Minimum and maximum contig length. Caps value from NX filtering.

min_cont_length_cutoff: 2250 # 500 max_cont_length_cutoff: 2250 # 1000

Minimum and maximum length of contigs containing CheckM markers. Caps value from NX filtering (initial value is NX_value / 2).

min_cont_length_cutoff_marker: 2250 # 250 max_cont_length_cutoff_marker: 2250 # 400

Maximum number of contigs binny uses. If the number of available

contigs after minimum size filtering exceeds this, binny will

increase the minimum size threshold until the maximum is reached.

Prevents use of excessive amounts of memory on large assemblies.

Default should ensure adequate performance, adjust e.g. according

to available memory.

max_n_contigs: 5.0e5 # 3.5e5

Maximum marker set lineage depth to check bin quality with:

0: 'domain', 1: 'phylum', 2: 'class', 3: 'order', 4: 'family', 5: 'genus', 6: 'species'

max_marker_lineage_depth_lvl: 2

Distance metric for opentSNE and HDBSCAN.

distance_metric: 'manhattan' embedding:

Maximum number of binny iterations.

max_iterations: 50 clustering:

Increasing the HDBSCAN cluster selection epsilon beyond 0.5

is not advised as it might massively increase run time, but it might

help recover fragmented genomes that would be missed with lower settings.

hdbscan_epsilon_range: '0.250,0.000'

Adapted from the HDBSCAN manual: 'Measure of how conservative the

clustering should be. With larger values, more points will be declared

as noise, and clusters will be restricted to progressively more dense areas.'.

hdbscan_min_samples_range: '1,5,10' # '1,2,4,6,8'

Use depth as additional dimension during the initial clustering.

include_depth_initial: 'False'

Use depth as additional dimension during the main clusterings.

include_depth_main: 'False' bin_quality:

Minimum value binny will lower completeness to while running.

min_completeness: 72.5

Completeness threshold binny wilt begin with.

start_completeness: 92.5

Minimum purity for bins to be selected.

ohickl commented 1 year ago

Hi Nathan,

it could be a problem limiting the maximum number of cores. Could you run sed -i -e 's/getThreads(80)/getThreads(64)/g' Snakefile in the binny dir and try again?

Best

Oskar

Nathhub commented 1 year ago

Hi Oskar,

Thanks for your answer and suggestion.

I have tried changing getThreads(80) to getThreads(64) as you suggested but the job still gets stopped at job 9 with the same error... Any other ideas on how to solve this problem?

Cheers

Nathhub commented 1 year ago

Hi Oskar,

I have tried again to run binny but this time reducing the number of threads to 40, using the command: ./binny -c -r -t 40 config/config.default.yaml and this time it worked.

Thanks