Error: Process failed with retcode -13

[eeaunin]

Hello, Some of my runs of FCS-GX have crashed with the message Error: Process failed with retcode -13. The log of a run like that is below.

=============================================================================== 
Source:      /mft-volume 
Destination: /app/db/gxdb 
Resuming failed transfer in /app/db/gxdb... 
Space check: Available:1.18TiB; Existing:0B; Incoming:464.34GiB; Delta:464.34GiB

Requires transfer: 59B all.meta.jsonl 
Copying /mft-volume/all.meta.jsonl to /app/db/gxdb/all.meta.jsonl.part... 

Requires transfer: 187B all.README.txt 
Copying /mft-volume/all.README.txt to /app/db/gxdb/all.README.txt.part... 

Requires transfer: 6.09MiB all.taxa.tsv 
Copying /mft-volume/all.taxa.tsv to /app/db/gxdb/all.taxa.tsv.part... 

Requires transfer: 7.86MiB all.blast_div.tsv.gz 
Copying /mft-volume/all.blast_div.tsv.gz to /app/db/gxdb/all.blast_div.tsv.gz.part... 

Requires transfer: 8.48MiB all.assemblies.tsv 
Copying /mft-volume/all.assemblies.tsv to /app/db/gxdb/all.assemblies.tsv.part... 

Requires transfer: 21.51MiB all.seq_info.tsv.gz 
Copying /mft-volume/all.seq_info.tsv.gz to /app/db/gxdb/all.seq_info.tsv.gz.part... 

Requires transfer: 165.14GiB all.gxs 
Copying /mft-volume/all.gxs to /app/db/gxdb/all.gxs.part... 

Requires transfer: 299.16GiB all.gxi 
Copying /mft-volume/all.gxi to /app/db/gxdb/all.gxi.part... 
Done. 
-----------------------------------------------------------------------------

tax-id    : 6318
fasta     : /sample-volume/assembly.fasta
size      : 297.4 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/ea10_gx_mapper_18426/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^6318\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/ea10_gx_mapper_18426/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^6318\t']
BLAST-div : nematodes
gx-div    : anml:nematodes
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/ea10_gx_mapper_18426/all.gxi
gx-ver    : Nov 27 2023 11:05:36; git:v0.5.0+branch--HEAD
output    : /output-volume//assembly.6318.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/assembly.fasta', tax_id=6318, species=None, split_fasta=True, div='anml:nematodes', gx_db='/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, ignore_same_kingdom=False, out_basename='/output-volume//assembly.6318', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, gzip_c='gzip -c', out_taxonomy_rpt='/output-volume//assembly.6318.taxonomy.rpt') 

####### Starting process ['cat', '/sample-volume/assembly.fasta']
####### Starting process ['gzip', '-cdf']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=311856885']
####### Starting process ['cat', '/sample-volume/assembly.fasta']
####### Starting process ['gzip', '-cdf']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--repeats-basis-fa=/dev/fd/5']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--output=/output-volume//assembly.6318.taxonomy.rpt.tmp', '--asserted-div=anml:nematodes', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']

-----------------------------------------------------------------------------

tax-id    : 6318
fasta     : /sample-volume/assembly.fasta
size      : 297.4 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/ea10_gx_mapper_18426/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^6318\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/ea10_gx_mapper_18426/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^6318\t']
BLAST-div : nematodes
gx-div    : anml:nematodes
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/ea10_gx_mapper_18426/all.gxi
gx-ver    : Nov 27 2023 11:05:36; git:v0.5.0+branch--HEAD
output    : /output-volume//assembly.6318.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/assembly.fasta', tax_id=6318, species=None, split_fasta=True, div='anml:nematodes', gx_db='/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, ignore_same_kingdom=False, out_basename='/output-volume//assembly.6318', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, gzip_c='gzip -c', out_taxonomy_rpt='/output-volume//assembly.6318.taxonomy.rpt') 

####### Starting process ['cat', '/sample-volume/assembly.fasta']
####### Starting process ['gzip', '-cdf']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=311856885']
####### Starting process ['cat', '/sample-volume/assembly.fasta']
####### Starting process ['gzip', '-cdf']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--repeats-basis-fa=/dev/fd/5']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--output=/output-volume//assembly.6318.taxonomy.rpt.tmp', '--asserted-div=anml:nematodes', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']

    GX requires the database to be entirely in RAM to avoid thrashing.
    Consider placing the database files in a non-swappable tmpfs or ramfs.
    See https://github.com/ncbi/fcs/wiki/FCS-GX for details.
    Will prefetch (vmtouch) the database pages to have the OS cache them in main memory.

        Prefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 1%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 2%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 3%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 4%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 5%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 6%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 7%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 8%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 9%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 10%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 11%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 12%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 13%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 14%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 15%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 16%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 17%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 18%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 19%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 20%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 21%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 22%...                         ^MPrefetching /app/db/gxdb/ea10_gx_mapper_18426/all.gxs 23%...                         ^MCommand terminated by signal 9
        Command being timed: "nice -n19 /app/bin/gx align --gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi --repeats-basis-fa=/dev/fd/5"
        User time (seconds): 2.41
        System time (seconds): 33.48
        Percent of CPU this job got: 6%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 9m 28.41s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 136074032
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 1068360
        Minor (reclaiming a frame) page faults: 7398795
        Voluntary context switches: 1079819
        Involuntary context switches: 1654
        Swaps: 0
        File system inputs: 67846334
        File system outputs: 224
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
Warning: missing header '##[["GX hits",2,1]]'
####### Cleaning up process ['cat', '/sample-volume/assembly.fasta']
Error: Process failed with retcode -13: ['cat', '/sample-volume/assembly.fasta'])
####### Cleaning up process ['gzip', '-cdf']
Error: Process failed with retcode -13: ['gzip', '-cdf'])
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
Error: Process failed with retcode -13: ['/app/bin/gx', 'split-fasta'])
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=311856885']
####### Cleaning up process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--repeats-basis-fa=/dev/fd/5']
Error: Process failed with retcode 9: ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--repeats-basis-fa=/dev/fd/5'])
####### Cleaning up process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--output=/output-volume//assembly.6318.taxonomy.rpt.tmp', '--asserted-div=anml:nematodes', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
####### Cleaning up process ['cat', '/sample-volume/assembly.fasta']
Error: Process failed with retcode -13: ['cat', '/sample-volume/assembly.fasta'])
####### Cleaning up process ['gzip', '-cdf']
Error: Process failed with retcode -13: ['gzip', '-cdf'])

-----------------------------------------------------------------------------

This run was done with a small assembly (assembly size: 12939844 bp, 261 contigs, average contig size: 49577.95 bp, largest contig: 301978 bp).

OS: Ubuntu 18.04.6 LTS Singularity: v3.10.0 FCS image: 0.5.0 Python: 3.10.11

Platform: LSF

Do you know what these error messages mean and what caused the crash here?

[etvedte]

It appears that the parent command was killed with signal 9, similar to #69 : Error: Process failed with retcode 9: ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/ea10_gx_mapper_18426/all.gxi', '--repeats-basis-fa=/dev/fd/5'])