search

ncbi / fcs

Foreign Contamination Screening caller scripts and documentation
Other
100 stars 13 forks source link

[BUG]: Long runtime with excess RAM/Memory and cores #97

Closed Artifice120 closed 2 weeks ago

Artifice120 commented 2 weeks ago

Describe the bug

While running the initial screening on a cluster node with 2,000Gb of ram and 56 cores I am still unable to finish screening a 600Mbp assembly within the 24 hour time limit. I made an Env file to specify 45 cpu's but when I check the actual used resources its only a few hundred mb of ram being used and 0.65 cpu's

To Reproduce

Slurm script used


#!/bin/bash
#SBATCH -J FCS
#SBATCH --account ACF-UTK0011
#SBATCH --time 24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=48
#SBATCH --partition=campus-bigmem
#SBATCH --mem=1500gb
#SBATCH --output=log/FCS.o%j
#SBATCH --error=error/FCS.e%j
#SBATCH --exclusive
#SBATCH --qos=campus-bigmem

source /lustre/isaac/scratch/jtorre28/enviro/mamba/micro/etc/profile.d/micromamba.sh

micromamba activate /lustre/isaac/scratch/jtorre28/enviro/mamba/micro/envs/python3.8

cd /lustre/isaac/scratch/jtorre28/fcs/

python3.8 ./fcs.py --image fcs-gx.sif --env-file env.txt screen genome --tax-id 202456 --debug --fasta ../foxgloves/purged/purged2/artho-only/pilon-bubble-filter-Arthro-only.fasta --out-dir ./gx_out/ --gx-db "/lustre/isaac/scratch/jtorre28/fcs/gxdb"

> **Here is the debug/log file **

INFO: /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

tax-id : 202456 fasta : /sample-volume/pilon-bubble-filter-Arthro-only.fasta size : 647.64 MiB split-fa : True ####### Starting process ['zcat', '-f', '/app/db/gxdb/gxdb/all.blast_div.tsv.gz'] ####### Starting process ['grep', '-E', '^202456\t'] ####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/gxdb/all.blast_div.tsv.gz'] ####### Cleaning up process ['grep', '-E', '^202456\t'] BLAST-div : aphids gx-div : anml:insects w/same-tax: True bin-dir : /app/bin gx-db : /app/db/gxdb/gxdb/all.gxi gx-ver : Jun 18 2024 11:01:15; git:v0.5.4-8-g3c7c426 output : /output-volume//pilon-bubble-filter-Arthro-only.202456.taxonomy.rpt

####### args: Namespace(fasta='/sample-volume/pilon-bubble-filter-Arthro-only.fasta', tax_id=202456, species=None, split_fasta=True, div='anml:insects', gx_db='/app/db/gxdb/gxdb/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, out_basename='/output-volume//pilon-bubble-filter-Arthro-only.202456', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, ignore_same_kingdom=None, out_taxonomy_rpt='/output-volume//pilon-bubble-filter-Arthro-only.202456.taxonomy.rpt')

####### CPU count: 56 ####### /proc/meminfo: MemTotal: 2112723736 kB MemFree: 2045109600 kB MemAvailable: 2084703752 kB Buffers: 1364264 kB Cached: 46325816 kB SwapCached: 0 kB Active: 9933652 kB Inactive: 38247504 kB Active(anon): 1271348 kB Inactive(anon): 1122640 kB Active(file): 8662304 kB Inactive(file): 37124864 kB Unevictable: 3072 kB Mlocked: 0 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 4 kB Writeback: 0 kB AnonPages: 481808 kB Mapped: 328124 kB Shmem: 1963280 kB KReclaimable: 2847868 kB Slab: 17460452 kB SReclaimable: 2847868 kB SUnreclaim: 14612584 kB KernelStack: 39376 kB PageTables: 17068 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 1060556168 kB Committed_AS: 5409932 kB VmallocTotal: 13743895347199 kB VmallocUsed: 0 kB VmallocChunk: 0 kB Percpu: 57344 kB HardwareCorrupted: 0 kB AnonHugePages: 200704 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB FileHugePages: 0 kB FilePmdMapped: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 9363428 kB DirectMap2M: 654872576 kB DirectMap1G: 1484783616 kB

####### top: Mem: 67614860K used, 2045108876K free, 1963280K shrd, 1364264K buff, 46326436K cached CPU: 0.1% usr 0.1% sys 0.0% nic 99.6% idle 0.0% io 0.0% irq 0.0% sirq Load average: 0.39 0.48 0.60 3/2405 1932200 PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND 19321991932198 jtorre28 R 1392 0.0 17 0.1 top -b 2304965 4734 nobody S 14.7g 0.0 8 0.0 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

####### Starting process ['cat', '/sample-volume/pilon-bubble-filter-Arthro-only.fasta'] ####### Starting process ['/app/bin/gx', 'get-fasta-stats'] ####### Starting process ['tee', '/dev/fd/6'] ####### Starting process ['/app/bin/gx', 'split-fasta'] ####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=679107392', '--buffer-size=104857600'] ####### Starting process ['cat', '/sample-volume/pilon-bubble-filter-Arthro-only.fasta'] ####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3'] ####### Starting process ['pv', '--quiet', '--buffer-size=104857600'] ####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/gxdb/all.gxi', '--output=/output-volume//pilon-bubble-filter-Arthro-only.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv'] Collecting masking statistics... Using GX_NUM_CORES=45 Collected masking stats: 0.670587 Gbp; 5.1631s; 129.805 Mbp/s. Baseline: 1.75809

Using GX_NUM_CORES=45


### Software versions  
 - Docker or Singularity version 1.1.8-1el8
 - Singularity FCS image version  0.5.4
 - Python version 3.8

### Additional context
From what I have seen in the tmp files, it appears to cover 39 contigs an hour out of 7020 contigs total is 43 lines long after an hour. 

real-time view resources during run

(python3) [jtorre28@login1 fcs]$ scontrol show node ilm0833 NodeName=ilm0833 Arch=x86_64 CoresPerSocket=28 CPUAlloc=56 CPUEfctv=56 CPUTot=56 CPULoad=0.65 AvailableFeatures=intel,icelake,avx512 ActiveFeatures=intel,icelake,avx512 Gres=(null) NodeAddr=ilm0833 NodeHostName=ilm0833 Version=23.02.2 OS=Linux 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 RealMemory=2016000 AllocMem=1536000 FreeMem=1992991 Sockets=2 Boards=1 State=ALLOCATED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=campus-bigmem BootTime=2024-07-08T09:15:50 SlurmdStartTime=2024-07-08T09:16:05 LastBusyTime=2024-09-09T14:44:06 ResumeAfterTime=None CfgTRES=cpu=56,mem=2016000M,billing=56 AllocTRES=cpu=56,mem=1500G CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s

Artifice120 commented 2 weeks ago

'top' command of running job

image

etvedte commented 2 weeks ago

Hello,

Our devs have reviewed the memory info in the debug logs, and there isn't anything that stands out to explain the slow runtime.

Can you add GX_PREFETCH=2 to your env.txt file and then run fcs.py --debug on a subset of your FASTA, say 10 Mbp? This will produce additional debug output that may help to diagnose the issue.

Artifice120 commented 2 weeks ago

After an hour of runtime.

The log file (1hour in) 

[jtorre28@login1 error]$ slide
INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

-----------------------------------------------------------------------------

tax-id    : 202456
fasta     : /sample-volume/test.fasta
size      : 9.53 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^202456\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^202456\t']
BLAST-div : aphids
gx-div    : anml:insects
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/all.gxi
gx-ver    : Jun 18 2024 11:01:15; git:v0.5.4-8-g3c7c426
output    : /output-volume//test.202456.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/test.fasta', tax_id=202456, species=None, split_fasta=True, div='anml:insects', gx_db='/app/db/gxdb/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, out_basename='/output-volume//test.202456', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, ignore_same_kingdom=None, out_taxonomy_rpt='/output-volume//test.202456.taxonomy.rpt')

#######
CPU count:  56
#######
/proc/meminfo:
MemTotal:       2112723736 kB
MemFree:        2047520684 kB
MemAvailable:   2084715788 kB
Buffers:         1365352 kB
Cached:         43928332 kB
SwapCached:            0 kB
Active:          9282344 kB
Inactive:       36504300 kB
Active(anon):    1284796 kB
Inactive(anon):  1143828 kB
Active(file):    7997548 kB
Inactive(file): 35360472 kB
Unevictable:        3072 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:              3028 kB
Writeback:             0 kB
AnonPages:        483488 kB
Mapped:           358648 kB
Shmem:           1996048 kB
KReclaimable:    2908072 kB
Slab:           17442512 kB
SReclaimable:    2908072 kB
SUnreclaim:     14534440 kB
KernelStack:       39392 kB
PageTables:        16900 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1060556168 kB
Committed_AS:    5222808 kB
VmallocTotal:   13743895347199 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            57344 kB
HardwareCorrupted:     0 kB
AnonHugePages:    200704 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     9777124 kB
DirectMap2M:    697450496 kB
DirectMap1G:    1441792000 kB

#######
top:
Mem: 65203672K used, 2047520064K free, 1996048K shrd, 1365352K buff, 43928588K cached
CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.59 0.85 0.76 3/2408 2170633
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
21706322170631 jtorre28 R     1392  0.0   9  0.3 top -b
2304965  4734 nobody   S    14.7g  0.0  20  0.0 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/app/bin/gx', 'get-fasta-stats']
####### Starting process ['tee', '/dev/fd/6']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
####### Starting process ['pv', '--quiet', '--buffer-size=104857600']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/all.gxi', '--output=/output-volume//test.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
Using GX_PREFETCH=2

/app/db/gxdb/all.gxs is 100% in RAM.
Prefetching /app/db/gxdb/all.gxs 99%...
Prefetched /app/db/gxdb/all.gxs in 295.673s; 0.599707 GB/s. The file is 100% in RAM.

/app/db/gxdb/all.gxi is 100% in RAM.
Prefetching /app/db/gxdb/all.gxi 99%...
Prefetched /app/db/gxdb/all.gxi in 884.524s; 0.363152 GB/s. The file is 100% in RAM.
Collecting masking statistics...
Using GX_NUM_CORES=45
Collected masking stats:  0.0098744 Gbp; 2.25409s; 4.38064 Mbp/s. Baseline: 1.0432

Using GX_NUM_CORES=45
####### Cleaning up process ['cat', '/sample-volume/test.fasta']
####### Cleaning up process ['tee', '/dev/fd/6']
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']

The top

top - 14:24:00 up 64 days,  5:08,  2 users,  load average: 0.34, 0.44, 0.63
Tasks: 859 total,   1 running, 856 sleeping,   0 stopped,   2 zombie
%Cpu(s):  0.1 us,  0.9 sy,  0.0 ni, 98.9 id,  0.0 wa,  0.1 hi,  0.0 si,  0.0 st
MiB Mem : 2063206.+total, 1995147.+free,  19115.8 used,  48943.5 buff/cache
MiB Swap:   4096.0 total,   4096.0 free,      0.0 used. 2033443.+avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2170641 jtorre28  39  19  468.7g  21.3g  19.9g S  29.0   1.1  19:35.62 gx
   4700 root      20   0  727576  39440  12020 S  11.6   0.0 990:44.73 node_exporter
   6170 root      20   0       0      0      0 S  11.6   0.0 865:56.26 kiblnd_sd_01_00
   6172 root      20   0       0      0      0 S  11.6   0.0 866:02.13 kiblnd_sd_01_02
   6173 root      20   0       0      0      0 S  11.6   0.0 866:00.19 kiblnd_sd_01_03
   6171 root      20   0       0      0      0 S  11.2   0.0 866:21.88 kiblnd_sd_01_01
   6220 root      20   0       0      0      0 S   1.0   0.0 145:43.24 ptlrpcd_01_04
   6234 root      20   0       0      0      0 S   1.0   0.0 147:05.68 ptlrpcd_01_18
   6236 root      20   0       0      0      0 S   1.0   0.0 146:02.93 ptlrpcd_01_20
   6238 root      20   0       0      0      0 S   1.0   0.0 145:23.16 ptlrpcd_01_22
2170625 jtorre28  20   0   45212  31372   9516 S   1.0   0.0   0:25.75 python3.11
   6189 root      20   0       0      0      0 S   0.7   0.0 137:17.27 ptlrpcd_00_01
   6190 root      20   0       0      0      0 S   0.7   0.0 152:20.07 ptlrpcd_00_02
   6191 root      20   0       0      0      0 S   0.7   0.0 136:14.81 ptlrpcd_00_03
   6192 root      20   0       0      0      0 S   0.7   0.0 150:40.67 ptlrpcd_00_04
   6193 root      20   0       0      0      0 S   0.7   0.0 137:41.76 ptlrpcd_00_05
   6194 root      20   0       0      0      0 S   0.7   0.0 151:24.60 ptlrpcd_00_06
   6195 root      20   0       0      0      0 S   0.7   0.0 137:23.16 ptlrpcd_00_07
   6197 root      20   0       0      0      0 S   0.7   0.0 137:30.95 ptlrpcd_00_09
   6198 root      20   0       0      0      0 S   0.7   0.0 151:24.12 ptlrpcd_00_10
   6199 root      20   0       0      0      0 S   0.7   0.0 137:16.93 ptlrpcd_00_11
   6200 root      20   0       0      0      0 S   0.7   0.0 152:14.00 ptlrpcd_00_12
   6201 root      20   0       0      0      0 S   0.7   0.0 136:40.31 ptlrpcd_00_13
   6202 root      20   0       0      0      0 S   0.7   0.0 151:05.22 ptlrpcd_00_14
   6203 root      20   0       0      0      0 S   0.7   0.0 137:45.53 ptlrpcd_00_15
   6204 root      20   0       0      0      0 S   0.7   0.0 151:26.70 ptlrpcd_00_16
   6206 root      20   0       0      0      0 S   0.7   0.0 150:45.26 ptlrpcd_00_18
   6207 root      20   0       0      0      0 S   0.7   0.0 137:48.72 ptlrpcd_00_19
   6208 root      20   0       0      0      0 S   0.7   0.0 151:42.94 ptlrpcd_00_20
   6210 root      20   0       0      0      0 S   0.7   0.0 152:00.60 ptlrpcd_00_22
   6211 root      20   0       0      0      0 S   0.7   0.0 136:59.20 ptlrpcd_00_23
   6212 root      20   0       0      0      0 S   0.7   0.0 151:15.15 ptlrpcd_00_24
   6213 root      20   0       0      0      0 S   0.7   0.0 137:26.41 ptlrpcd_00_25
   6214 root      20   0       0      0      0 S   0.7   0.0 151:26.72 ptlrpcd_00_26
   6215 root      20   0       0      0      0 S   0.7   0.0 137:21.01 ptlrpcd_00_27
   6216 root      20   0       0      0      0 S   0.7   0.0 145:56.90 ptlrpcd_01_00
   6218 root      20   0       0      0      0 S   0.7   0.0 146:25.75 ptlrpcd_01_02
   6219 root      20   0       0      0      0 S   0.7   0.0 130:36.48 ptlrpcd_01_03
   6221 root      20   0       0      0      0 S   0.7   0.0 131:04.20 ptlrpcd_01_05
   6222 root      20   0       0      0      0 S   0.7   0.0 145:43.50 ptlrpcd_01_06
   6223 root      20   0       0      0      0 S   0.7   0.0 130:52.56 ptlrpcd_01_07
   6224 root      20   0       0      0      0 S   0.7   0.0 146:03.57 ptlrpcd_01_08
   6225 root      20   0       0      0      0 S   0.7   0.0 130:31.25 ptlrpcd_01_09
   6226 root      20   0       0      0      0 S   0.7   0.0 146:09.94 ptlrpcd_01_10

The Pre-fetch portion took ~25 minutes.

Artifice120 commented 2 weeks ago

Subset has finished

Log File

[jtorre28@login1 error]$ slide
INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

-----------------------------------------------------------------------------

tax-id    : 202456
fasta     : /sample-volume/test.fasta
size      : 9.53 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^202456\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^202456\t']
BLAST-div : aphids
gx-div    : anml:insects
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/all.gxi
gx-ver    : Jun 18 2024 11:01:15; git:v0.5.4-8-g3c7c426
output    : /output-volume//test.202456.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/test.fasta', tax_id=202456, species=None, split_fasta=True, div='anml:insects', gx_db='/app/db/gxdb/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, out_basename='/output-volume//test.202456', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, ignore_same_kingdom=None, out_taxonomy_rpt='/output-volume//test.202456.taxonomy.rpt')

#######
CPU count:  56
#######
/proc/meminfo:
MemTotal:       2112723736 kB
MemFree:        2047520684 kB
MemAvailable:   2084715788 kB
Buffers:         1365352 kB
Cached:         43928332 kB
SwapCached:            0 kB
Active:          9282344 kB
Inactive:       36504300 kB
Active(anon):    1284796 kB
Inactive(anon):  1143828 kB
Active(file):    7997548 kB
Inactive(file): 35360472 kB
Unevictable:        3072 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:              3028 kB
Writeback:             0 kB
AnonPages:        483488 kB
Mapped:           358648 kB
Shmem:           1996048 kB
KReclaimable:    2908072 kB
Slab:           17442512 kB
SReclaimable:    2908072 kB
SUnreclaim:     14534440 kB
KernelStack:       39392 kB
PageTables:        16900 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1060556168 kB
Committed_AS:    5222808 kB
VmallocTotal:   13743895347199 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            57344 kB
HardwareCorrupted:     0 kB
AnonHugePages:    200704 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     9777124 kB
DirectMap2M:    697450496 kB
DirectMap1G:    1441792000 kB

#######
top:
Mem: 65203672K used, 2047520064K free, 1996048K shrd, 1365352K buff, 43928588K cached
CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.59 0.85 0.76 3/2408 2170633
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
21706322170631 jtorre28 R     1392  0.0   9  0.3 top -b
2304965  4734 nobody   S    14.7g  0.0  20  0.0 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/app/bin/gx', 'get-fasta-stats']
####### Starting process ['tee', '/dev/fd/6']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
####### Starting process ['pv', '--quiet', '--buffer-size=104857600']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/all.gxi', '--output=/output-volume//test.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
Using GX_PREFETCH=2

/app/db/gxdb/all.gxs is 100% in RAM.
Prefetching /app/db/gxdb/all.gxs 99%...
Prefetched /app/db/gxdb/all.gxs in 295.673s; 0.599707 GB/s. The file is 100% in RAM.

/app/db/gxdb/all.gxi is 100% in RAM.
Prefetching /app/db/gxdb/all.gxi 99%...
Prefetched /app/db/gxdb/all.gxi in 884.524s; 0.363152 GB/s. The file is 100% in RAM.
Collecting masking statistics...
Using GX_NUM_CORES=45
Collected masking stats:  0.0098744 Gbp; 2.25409s; 4.38064 Mbp/s. Baseline: 1.0432

Using GX_NUM_CORES=45
####### Cleaning up process ['cat', '/sample-volume/test.fasta']
####### Cleaning up process ['tee', '/dev/fd/6']
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
^C
[jtorre28@login1 error]$ eyeball
total 64K
 20K -rw-r--r-- 1 jtorre28 tug2106 18K Sep 10 13:44 FCS.e2436464
4.0K -rw-r--r-- 1 jtorre28 tug2106 156 Aug 13 09:41 busco.e2316252
   0 -rw-r--r-- 1 jtorre28 tug2106   0 Aug 12 16:13 jelly.e2308931
 36K -rw-r--r-- 1 jtorre28 tug2106 31K Aug  7 12:56 jvc.e2281882
4.0K -rw-r--r-- 1 jtorre28 tug2106  87 Jul 26 16:41 NCBI-egap.e2069274
   0 -rwxrwxrwx 1 jtorre28 tug2106   0 Jul  5 10:12 breaker.e1697953
[jtorre28@login1 error]$ slide
INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

-----------------------------------------------------------------------------

tax-id    : 202456
fasta     : /sample-volume/test.fasta
size      : 9.53 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^202456\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^202456\t']
BLAST-div : aphids
gx-div    : anml:insects
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/all.gxi
gx-ver    : Jun 18 2024 11:01:15; git:v0.5.4-8-g3c7c426
output    : /output-volume//test.202456.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/test.fasta', tax_id=202456, species=None, split_fasta=True, div='anml:insects', gx_db='/app/db/gxdb/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, out_basename='/output-volume//test.202456', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, ignore_same_kingdom=None, out_taxonomy_rpt='/output-volume//test.202456.taxonomy.rpt')

#######
CPU count:  56
#######
/proc/meminfo:
MemTotal:       2112723736 kB
MemFree:        2047520684 kB
MemAvailable:   2084715788 kB
Buffers:         1365352 kB
Cached:         43928332 kB
SwapCached:            0 kB
Active:          9282344 kB
Inactive:       36504300 kB
Active(anon):    1284796 kB
Inactive(anon):  1143828 kB
Active(file):    7997548 kB
Inactive(file): 35360472 kB
Unevictable:        3072 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:              3028 kB
Writeback:             0 kB
AnonPages:        483488 kB
Mapped:           358648 kB
Shmem:           1996048 kB
KReclaimable:    2908072 kB
Slab:           17442512 kB
SReclaimable:    2908072 kB
SUnreclaim:     14534440 kB
KernelStack:       39392 kB
PageTables:        16900 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1060556168 kB
Committed_AS:    5222808 kB
VmallocTotal:   13743895347199 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            57344 kB
HardwareCorrupted:     0 kB
AnonHugePages:    200704 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     9777124 kB
DirectMap2M:    697450496 kB
DirectMap1G:    1441792000 kB

#######
top:
Mem: 65203672K used, 2047520064K free, 1996048K shrd, 1365352K buff, 43928588K cached
CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.59 0.85 0.76 3/2408 2170633
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
21706322170631 jtorre28 R     1392  0.0   9  0.3 top -b
2304965  4734 nobody   S    14.7g  0.0  20  0.0 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/app/bin/gx', 'get-fasta-stats']
####### Starting process ['tee', '/dev/fd/6']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
####### Starting process ['pv', '--quiet', '--buffer-size=104857600']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/all.gxi', '--output=/output-volume//test.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
Using GX_PREFETCH=2

/app/db/gxdb/all.gxs is 100% in RAM.
Prefetching /app/db/gxdb/all.gxs 99%...
Prefetched /app/db/gxdb/all.gxs in 295.673s; 0.599707 GB/s. The file is 100% in RAM.

/app/db/gxdb/all.gxi is 100% in RAM.
Prefetching /app/db/gxdb/all.gxi 99%...
Prefetched /app/db/gxdb/all.gxi in 884.524s; 0.363152 GB/s. The file is 100% in RAM.
Collecting masking statistics...
Using GX_NUM_CORES=45
Collected masking stats:  0.0098744 Gbp; 2.25409s; 4.38064 Mbp/s. Baseline: 1.0432

Using GX_NUM_CORES=45
####### Cleaning up process ['cat', '/sample-volume/test.fasta']
####### Cleaning up process ['tee', '/dev/fd/6']
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
^C
[jtorre28@login1 error]$ pwd
/lustre/isaac/scratch/jtorre28/error
[jtorre28@login1 error]$ slide
INFO:    /etc/singularity/ exists; cleanup by system administrator is not complete (see https://apptainer.org/docs/admin/latest/singularity_migration.html)

-----------------------------------------------------------------------------

tax-id    : 202456
fasta     : /sample-volume/test.fasta
size      : 9.53 MiB
split-fa  : True
####### Starting process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Starting process ['grep', '-E', '^202456\t']
####### Cleaning up process ['zcat', '-f', '/app/db/gxdb/all.blast_div.tsv.gz']
####### Cleaning up process ['grep', '-E', '^202456\t']
BLAST-div : aphids
gx-div    : anml:insects
w/same-tax: True
bin-dir   : /app/bin
gx-db     : /app/db/gxdb/all.gxi
gx-ver    : Jun 18 2024 11:01:15; git:v0.5.4-8-g3c7c426
output    : /output-volume//test.202456.taxonomy.rpt

-----------------------------------------------------------------------------

####### args: Namespace(fasta='/sample-volume/test.fasta', tax_id=202456, species=None, split_fasta=True, div='anml:insects', gx_db='/app/db/gxdb/all.gxi', mask_transposons=None, bin_dir='/app/bin', allow_same_species=True, out_basename='/output-volume//test.202456', out_dir='/output-volume/', action_report=True, save_hits=False, generate_logfile=False, debug=True, phone_home_label=None, gc_acc=None, gc_genomes_root_dir=None, production_build_name=None, ignore_same_kingdom=None, out_taxonomy_rpt='/output-volume//test.202456.taxonomy.rpt')

#######
CPU count:  56
#######
/proc/meminfo:
MemTotal:       2112723736 kB
MemFree:        2047520684 kB
MemAvailable:   2084715788 kB
Buffers:         1365352 kB
Cached:         43928332 kB
SwapCached:            0 kB
Active:          9282344 kB
Inactive:       36504300 kB
Active(anon):    1284796 kB
Inactive(anon):  1143828 kB
Active(file):    7997548 kB
Inactive(file): 35360472 kB
Unevictable:        3072 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4194300 kB
Dirty:              3028 kB
Writeback:             0 kB
AnonPages:        483488 kB
Mapped:           358648 kB
Shmem:           1996048 kB
KReclaimable:    2908072 kB
Slab:           17442512 kB
SReclaimable:    2908072 kB
SUnreclaim:     14534440 kB
KernelStack:       39392 kB
PageTables:        16900 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    1060556168 kB
Committed_AS:    5222808 kB
VmallocTotal:   13743895347199 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
Percpu:            57344 kB
HardwareCorrupted:     0 kB
AnonHugePages:    200704 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     9777124 kB
DirectMap2M:    697450496 kB
DirectMap1G:    1441792000 kB

#######
top:
Mem: 65203672K used, 2047520064K free, 1996048K shrd, 1365352K buff, 43928588K cached
CPU:  0.1% usr  0.1% sys  0.0% nic 99.6% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.59 0.85 0.76 3/2408 2170633
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
21706322170631 jtorre28 R     1392  0.0   9  0.3 top -b
2304965  4734 nobody   S    14.7g  0.0  20  0.0 /usr/lib64/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/app/bin/gx', 'get-fasta-stats']
####### Starting process ['tee', '/dev/fd/6']
####### Starting process ['/app/bin/gx', 'split-fasta']
####### Starting process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
####### Starting process ['cat', '/sample-volume/test.fasta']
####### Starting process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
####### Starting process ['pv', '--quiet', '--buffer-size=104857600']
####### Starting process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/all.gxi', '--output=/output-volume//test.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
Using GX_PREFETCH=2

/app/db/gxdb/all.gxs is 100% in RAM.
Prefetching /app/db/gxdb/all.gxs 99%...
Prefetched /app/db/gxdb/all.gxs in 295.673s; 0.599707 GB/s. The file is 100% in RAM.

/app/db/gxdb/all.gxi is 100% in RAM.
Prefetching /app/db/gxdb/all.gxi 99%...
Prefetched /app/db/gxdb/all.gxi in 884.524s; 0.363152 GB/s. The file is 100% in RAM.
Collecting masking statistics...
Using GX_NUM_CORES=45
Collected masking stats:  0.0098744 Gbp; 2.25409s; 4.38064 Mbp/s. Baseline: 1.0432

Using GX_NUM_CORES=45
####### Cleaning up process ['cat', '/sample-volume/test.fasta']
####### Cleaning up process ['tee', '/dev/fd/6']
####### Cleaning up process ['/app/bin/gx', 'split-fasta']
####### Cleaning up process ['pv', '-Wbratpe', '--interval=0.5', '--size=10000300', '--buffer-size=104857600']
Processed 97 queries, 9.8802Mbp in 8080.21s. (0.00122277Mbp/s); num-jobs:103
        Command being timed: "nice -n19 /app/bin/gx align --gx-db=/app/db/gxdb/all.gxi --repeats-basis-fa=/dev/fd/3"
        User time (seconds): 169.54
        System time (seconds): 2390.73
        Percent of CPU this job got: 27%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2h 34m 27s
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 104984320
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 133666211
        Voluntary context switches: 10051157
        Involuntary context switches: 19156
        Swaps: 0
        File system inputs: 596521080
        File system outputs: 1656
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0
####### Cleaning up process ['/busybox/time', '-v', 'nice', '-n19', '/app/bin/gx', 'align', '--gx-db=/app/db/gxdb/all.gxi', '--repeats-basis-fa=/dev/fd/3']
####### Cleaning up process ['pv', '--quiet', '--buffer-size=104857600']
####### Cleaning up process ['/app/bin/gx', 'taxify', '--gx-db=/app/db/gxdb/all.gxi', '--output=/output-volume//test.202456.taxonomy.rpt.tmp', '--asserted-div=anml:insects', '--db-exclude-locs=/app/bin/db_exclude.locs.tsv']
####### Cleaning up process ['/app/bin/gx', 'get-fasta-stats']
####### Cleaning up process ['cat', '/sample-volume/test.fasta']
####### ['/app/bin/classify_taxonomy', '--in=/output-volume//test.202456.taxonomy.rpt.tmp', '--fasta-stats=/output-volume//test.202456.fasta-stats.json']
Species                    : None
Asserted div               : anml:insects
Inferred primary-divs      : ['anml:insects']
Corrected primary-divs     : ['anml:insects']
Putative contaminant divs  : ['prok:g-proteobacteria']
Aggregate coverage         : 90%
Minimum contam. coverage   : 20%
####### ['/app/bin/action_report', '--in=/output-volume//test.202456.taxonomy.rpt']

-----------------------------------------------------------------------------

fcs_gx_report.txt contamination summary:
----------------------------------------
                                seqs      bases
                               ----- ----------
TOTAL                             19     222075
-----                          ----- ----------
prok:g-proteobacteria             18     217766
virs:prokaryotic viruses           1       4309

-----------------------------------------------------------------------------

fcs_gx_report.txt action summary:
---------------------------------
                                seqs      bases
                               ----- ----------
TOTAL                             19     222075
-----                          ----- ----------
EXCLUDE                           16     203360
TRIM                               3      18715

-----------------------------------------------------------------------------

--------------------------------------------------------------------------------------------

NCBI collects limited usage data for each run of FCS by default. To disable usage reporting:

python3 fcs.py --no-report-analytics ...

or export NCBI_FCS_REPORT_ANALYTICS=1 to enable,
or export NCBI_FCS_REPORT_ANALYTICS=0 to disable the reporting.

Usage data collected by NCBI is documented in the FCS privacy notice
at https://github.com/ncbi/fcs/blob/main/PRIVACY.md

--------------------------------------------------------------------------------------------
etvedte commented 2 weeks ago

We are hypothesizing that the host you are running on has a policy to evict memory-mapped pages from RAM, resulting in disk-thrashing and slow down. Reason being that the cached memory value of the memoryinfo log reads Cached: 43928332 kB and should be >460 GiB if the database is cached in RAM. If you run the pipeline for successive genomes and the db is cached, the Cached: value should be much larger than 460.

If instead of top you run perf stat -e major-faults -I 1000 -p $(pidof -s gx) while it is running, you should be able to see any major-page-faults in real time that would suggest thrashing. 

perf stat -e major-faults -I 1000 -p $(pidof -s gx)
#           time             counts unit events
     1.000937737                  0      major-faults:u                                                      
     2.001936971                  0      major-faults:u                                                      
     3.003027731      <not counted>      major-faults:u                                                      
     4.003937076                  0      major-faults:u                                                      
     5.005938799                  0      major-faults:u                                                      
     6.006937688                  0      major-faults:u                                                      
     7.007936431                  0      major-faults:u

The counts column is expected to be 0. Can you try this?

etvedte commented 2 weeks ago

Looks like there were 0 major page faults here. We will look this over as well.

Artifice120 commented 2 weeks ago

Don't have root access so I created a python virtual environment to install perf-py. Seems to be different.

Artifice120 commented 2 weeks ago

How about this?

[jtorre28@clrm1219 ~]$ queue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           2439435 campus-bi      FCS jtorre28  R      54:49      1 clrm1219
[jtorre28@clrm1219 ~]$ ps -o min_flt,maj_flt $(pidof -s gx)
 MINFL  MAJFL
  5899      1

reran subset again

Artifice120 commented 2 weeks ago

Not sure if this helps.

Connected to the node while rerunning the full and found the PID with $(pidof -s gx) and opened /proc/$(pidof -s gx)/maps

[jtorre28@clrm1216 144892]$ cat maps
00400000-00776000 r-xp 00000000 00:33 5220                               /app/bin/gx
00975000-00985000 rwxp 00375000 00:33 5220                               /app/bin/gx
00985000-00a08000 rwxp 00000000 00:00 0
00fa1000-022c7000 rwxp 00000000 00:00 0                                  [heap]
7f59d46b0000-7f59d4fb1000 rwxp 00000000 00:00 0
7ffe84609000-7ffe8462c000 rwxp 00000000 00:00 0                          [stack]
7ffe8466e000-7ffe84672000 r--p 00000000 00:00 0                          [vvar]
7ffe84672000-7ffe84674000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Artifice120 commented 2 weeks ago 
00400000-00776000 r-xp 00000000 00:33 5220                               /app/bin/gx
00975000-00985000 rwxp 00375000 00:33 5220                               /app/bin/gx
00985000-00a08000 rwxp 00000000 00:00 0
0148d000-134ea000 rwxp 00000000 00:00 0                                  [heap]
7f4c3e575000-7f4c40000000 rwxp 00000000 00:00 0
7f4c40000000-7f4c40057000 rwxp 00000000 00:00 0
7f4c40057000-7f4c44000000 ---p 00000000 00:00 0
7f4c44000000-7f4c44021000 rwxp 00000000 00:00 0
7f4c44021000-7f4c48000000 ---p 00000000 00:00 0
7f4c48000000-7f4c480f3000 rwxp 00000000 00:00 0
7f4c480f3000-7f4c4c000000 ---p 00000000 00:00 0
7f4c4c000000-7f4c4c021000 rwxp 00000000 00:00 0
7f4c4c021000-7f4c50000000 ---p 00000000 00:00 0
7f4c50000000-7f4c50021000 rwxp 00000000 00:00 0
7f4c50021000-7f4c54000000 ---p 00000000 00:00 0
7f4c54000000-7f4c540a2000 rwxp 00000000 00:00 0
7f4c540a2000-7f4c58000000 ---p 00000000 00:00 0
7f4c58000000-7f4c580b5000 rwxp 00000000 00:00 0
7f4c580b5000-7f4c5c000000 ---p 00000000 00:00 0
7f4c5c10a000-7f4c5c671000 rwxp 00000000 00:00 0
7f4c5c671000-7f4c5c672000 ---p 00000000 00:00 0
7f4c5c672000-7f4c5ce72000 rwxp 00000000 00:00 0
7f4c5ce72000-7f4c5ce73000 ---p 00000000 00:00 0
7f4c5ce73000-7f4c5d7fb000 rwxp 00000000 00:00 0
7f4c5d7fb000-7f4c5d7fc000 ---p 00000000 00:00 0
7f4c5d7fc000-7f4c5dffc000 rwxp 00000000 00:00 0
7f4c5dffc000-7f4c5dffd000 ---p 00000000 00:00 0
7f4c5dffd000-7f4c5e7fd000 rwxp 00000000 00:00 0
7f4c5e7fd000-7f4c5e7fe000 ---p 00000000 00:00 0
7f4c5e7fe000-7f4c5effe000 rwxp 00000000 00:00 0
7f4c5effe000-7f4c5efff000 ---p 00000000 00:00 0
7f4c5efff000-7f4c5f7ff000 rwxp 00000000 00:00 0
7f4c5f7ff000-7f4c5f800000 ---p 00000000 00:00 0
7f4c5f800000-7f4c60000000 rwxp 00000000 00:00 0
7f4c60000000-7f4c60021000 rwxp 00000000 00:00 0
7f4c60021000-7f4c64000000 ---p 00000000 00:00 0
7f4c64000000-7f4c640b0000 rwxp 00000000 00:00 0
7f4c640b0000-7f4c68000000 ---p 00000000 00:00 0
7f4c68000000-7f4c68021000 rwxp 00000000 00:00 0
7f4c68021000-7f4c6c000000 ---p 00000000 00:00 0
7f4c6c000000-7f4c6c021000 rwxp 00000000 00:00 0
7f4c6c021000-7f4c70000000 ---p 00000000 00:00 0
7f4c70000000-7f4c70021000 rwxp 00000000 00:00 0
7f4c70021000-7f4c74000000 ---p 00000000 00:00 0
7f4c74000000-7f4c74021000 rwxp 00000000 00:00 0
7f4c74021000-7f4c78000000 ---p 00000000 00:00 0
7f4c78000000-7f4c78021000 rwxp 00000000 00:00 0
7f4c78021000-7f4c7c000000 ---p 00000000 00:00 0
7f4c7c051000-7f4c7c1d9000 rwxp 00000000 00:00 0
7f4c7c1d9000-7f4c7c1da000 ---p 00000000 00:00 0
7f4c7c1da000-7f4c7cb62000 rwxp 00000000 00:00 0
7f4c7cb62000-7f4c7cb63000 ---p 00000000 00:00 0
7f4c7cb63000-7f4c7d4eb000 rwxp 00000000 00:00 0
7f4c7d4eb000-7f4c7d4ec000 ---p 00000000 00:00 0
7f4c7d4ec000-7f4c7de74000 rwxp 00000000 00:00 0
7f4c7de74000-7f4c7de75000 ---p 00000000 00:00 0
7f4c7de75000-7f4c7e7fd000 rwxp 00000000 00:00 0
7f4c7e7fd000-7f4c7e7fe000 ---p 00000000 00:00 0
7f4c7e7fe000-7f4c7effe000 rwxp 00000000 00:00 0
7f4c7effe000-7f4c7efff000 ---p 00000000 00:00 0
7f4c7efff000-7f4c7f7ff000 rwxp 00000000 00:00 0
7f4c7f7ff000-7f4c7f800000 ---p 00000000 00:00 0
7f4c7f800000-7f4c80000000 rwxp 00000000 00:00 0
7f4c80000000-7f4c8027b000 rwxp 00000000 00:00 0
7f4c8027b000-7f4c84000000 ---p 00000000 00:00 0
7f4c84000000-7f4c84021000 rwxp 00000000 00:00 0
7f4c84021000-7f4c88000000 ---p 00000000 00:00 0
7f4c88000000-7f4c88021000 rwxp 00000000 00:00 0
7f4c88021000-7f4c8c000000 ---p 00000000 00:00 0
7f4c8c000000-7f4c8c052000 rwxp 00000000 00:00 0
7f4c8c052000-7f4c90000000 ---p 00000000 00:00 0
7f4c90000000-7f4c90188000 rwxp 00000000 00:00 0
7f4c90188000-7f4c94000000 ---p 00000000 00:00 0
7f4c94000000-7f4c9408d000 rwxp 00000000 00:00 0
7f4c9408d000-7f4c98000000 ---p 00000000 00:00 0
7f4c98000000-7f4c982a5000 rwxp 00000000 00:00 0
7f4c982a5000-7f4c9c000000 ---p 00000000 00:00 0
7f4c9c000000-7f4c9c021000 rwxp 00000000 00:00 0
7f4c9c021000-7f4ca0000000 ---p 00000000 00:00 0
7f4ca007a000-7f4ca0506000 rwxp 00000000 00:00 0
7f4ca0506000-7f4ca0507000 ---p 00000000 00:00 0
7f4ca0507000-7f4ca0d07000 rwxp 00000000 00:00 0
7f4ca0d07000-7f4ca0d08000 ---p 00000000 00:00 0
7f4ca0d08000-7f4ca1508000 rwxp 00000000 00:00 0
7f4ca1508000-7f4ca1509000 ---p 00000000 00:00 0
7f4ca1509000-7f4ca1e91000 rwxp 00000000 00:00 0
7f4ca1e91000-7f4ca1e92000 ---p 00000000 00:00 0
7f4ca1e92000-7f4ca2692000 rwxp 00000000 00:00 0
7f4ca2692000-7f4ca2693000 ---p 00000000 00:00 0
7f4ca2693000-7f4ca2e93000 rwxp 00000000 00:00 0
7f4ca2e93000-7f4ca2e94000 ---p 00000000 00:00 0
7f4ca2e94000-7f4ca3694000 rwxp 00000000 00:00 0
7f4ca3694000-7f4ca3695000 ---p 00000000 00:00 0
7f4ca3695000-7f4ca4000000 rwxp 00000000 00:00 0
7f4ca4000000-7f4ca4021000 rwxp 00000000 00:00 0
7f4ca4021000-7f4ca8000000 ---p 00000000 00:00 0
7f4ca8000000-7f4ca8021000 rwxp 00000000 00:00 0
7f4ca8021000-7f4cac000000 ---p 00000000 00:00 0
7f4cac000000-7f4cac021000 rwxp 00000000 00:00 0
7f4cac021000-7f4cb0000000 ---p 00000000 00:00 0
7f4cb0000000-7f4cb0021000 rwxp 00000000 00:00 0
7f4cb0021000-7f4cb4000000 ---p 00000000 00:00 0
7f4cb4000000-7f4cb4021000 rwxp 00000000 00:00 0
7f4cb4021000-7f4cb8000000 ---p 00000000 00:00 0
7f4cb8000000-7f4cb8021000 rwxp 00000000 00:00 0
7f4cb8021000-7f4cbc000000 ---p 00000000 00:00 0
7f4cbc000000-7f4cbc021000 rwxp 00000000 00:00 0
7f4cbc021000-7f4cc0000000 ---p 00000000 00:00 0
7f4cc0000000-7f4cc0047000 rwxp 00000000 00:00 0
7f4cc0047000-7f4cc4000000 ---p 00000000 00:00 0
7f4cc4000000-7f4cc41de000 rwxp 00000000 00:00 0
7f4cc41de000-7f4cc8000000 ---p 00000000 00:00 0
7f4cc8000000-7f4cc8055000 rwxp 00000000 00:00 0
7f4cc8055000-7f4ccc000000 ---p 00000000 00:00 0
7f4ccc000000-7f4ccc0dd000 rwxp 00000000 00:00 0
7f4ccc0dd000-7f4cd0000000 ---p 00000000 00:00 0
7f4cd0000000-7f4cd0021000 rwxp 00000000 00:00 0
7f4cd0021000-7f4cd4000000 ---p 00000000 00:00 0
7f4cd4000000-7f4cd40b3000 rwxp 00000000 00:00 0
7f4cd40b3000-7f4cd8000000 ---p 00000000 00:00 0
7f4cd8000000-7f4cd8131000 rwxp 00000000 00:00 0
7f4cd8131000-7f4cdc000000 ---p 00000000 00:00 0
7f4cdc000000-7f4cdc110000 rwxp 00000000 00:00 0
7f4cdc110000-7f4ce0000000 ---p 00000000 00:00 0
7f4ce0000000-7f4ce0021000 rwxp 00000000 00:00 0
7f4ce0021000-7f4ce4000000 ---p 00000000 00:00 0
7f4ce4000000-7f4ce40b9000 rwxp 00000000 00:00 0
7f4ce40b9000-7f4ce8000000 ---p 00000000 00:00 0
7f4ce8000000-7f4ce807f000 rwxp 00000000 00:00 0
7f4ce807f000-7f4cec000000 ---p 00000000 00:00 0
7f4cec000000-7f4cec021000 rwxp 00000000 00:00 0
7f4cec021000-7f4cf0000000 ---p 00000000 00:00 0
7f4cf0000000-7f4cf0021000 rwxp 00000000 00:00 0
7f4cf0021000-7f4cf4000000 ---p 00000000 00:00 0
7f4cf4000000-7f4cf4021000 rwxp 00000000 00:00 0
7f4cf4021000-7f4cf8000000 ---p 00000000 00:00 0
7f4cf8000000-7f4cf8131000 rwxp 00000000 00:00 0
7f4cf8131000-7f4cfc000000 ---p 00000000 00:00 0
7f4cfc04d000-7f4cfc1d5000 rwxp 00000000 00:00 0
7f4cfc1d5000-7f4cfc1d6000 ---p 00000000 00:00 0
7f4cfc1d6000-7f4cfc9d6000 rwxp 00000000 00:00 0
7f4cfc9d6000-7f4cfc9d7000 ---p 00000000 00:00 0
7f4cfc9d7000-7f4cfd35b000 rwxp 00000000 00:00 0
7f4cfd35b000-7f4cfd35c000 ---p 00000000 00:00 0
7f4cfd35c000-7f4cfdce4000 rwxp 00000000 00:00 0
7f4cfdce4000-7f4cfdce5000 ---p 00000000 00:00 0
7f4cfdce5000-7f4cfe66d000 rwxp 00000000 00:00 0
7f4cfe66d000-7f4cfe66e000 ---p 00000000 00:00 0
7f4cfe66e000-7f4cfeff6000 rwxp 00000000 00:00 0
7f4cfeff6000-7f4cfeff7000 ---p 00000000 00:00 0
7f4cfeff7000-7f4cff7f7000 rwxp 00000000 00:00 0
7f4cff7f7000-7f4cff7f8000 ---p 00000000 00:00 0
7f4cff7f8000-7f4cffff8000 rwxp 00000000 00:00 0
7f4cffff8000-7f4cffff9000 ---p 00000000 00:00 0
7f4cffff9000-7f4d007f9000 rwxp 00000000 00:00 0
7f4d007f9000-7f4d007fa000 ---p 00000000 00:00 0
7f4d007fa000-7f4d00ffa000 rwxp 00000000 00:00 0
7f4d00ffa000-7f4d00ffb000 ---p 00000000 00:00 0
7f4d00ffb000-7f4d017fb000 rwxp 00000000 00:00 0
7f4d017fb000-7f4d017fc000 ---p 00000000 00:00 0
7f4d017fc000-7f4d01ffc000 rwxp 00000000 00:00 0
7f4d01ffc000-7f4d01ffd000 ---p 00000000 00:00 0
7f4d01ffd000-7f4d027fd000 rwxp 00000000 00:00 0
7f4d027fd000-7f4d027fe000 ---p 00000000 00:00 0
7f4d027fe000-7f4d02ffe000 rwxp 00000000 00:00 0
7f4d02ffe000-7f4d02fff000 ---p 00000000 00:00 0
7f4d02fff000-7f4d037ff000 rwxp 00000000 00:00 0
7f4d037ff000-7f4d03800000 ---p 00000000 00:00 0
7f4d03800000-7f4d04000000 rwxp 00000000 00:00 0
7f4d04000000-7f4d04214000 rwxp 00000000 00:00 0
7f4d04214000-7f4d08000000 ---p 00000000 00:00 0
7f4d080ba000-7f4d08552000 rwxp 00000000 00:00 0
7f4d08552000-7f4d08553000 ---p 00000000 00:00 0
7f4d08553000-7f4d08d53000 rwxp 00000000 00:00 0
7f4d08d53000-7f4d08d54000 ---p 00000000 00:00 0
7f4d08d54000-7f4d09554000 rwxp 00000000 00:00 0
7f4d09554000-7f4d09555000 ---p 00000000 00:00 0
7f4d09555000-7f4d09d55000 rwxp 00000000 00:00 0
7f4d09d55000-7f4d09d56000 ---p 00000000 00:00 0
7f4d09d56000-7f4d0a6de000 rwxp 00000000 00:00 0
7f4d0a6de000-7f4d0a6df000 ---p 00000000 00:00 0
7f4d0a6df000-7f4d0aedf000 rwxp 00000000 00:00 0
7f4d0aedf000-7f4d0aee0000 ---p 00000000 00:00 0
7f4d0aee0000-7f4d0b864000 rwxp 00000000 00:00 0
7f4d0b864000-7f4d0b865000 ---p 00000000 00:00 0
7f4d0b865000-7f4d0c1ed000 rwxp 00000000 00:00 0
7f4d0c1ed000-7f4d0c1ee000 ---p 00000000 00:00 0
7f4d0c1ee000-7f4d0c9ee000 rwxp 00000000 00:00 0
7f4d0c9ee000-7f4d0c9ef000 ---p 00000000 00:00 0
7f4d0c9ef000-7f4d58c48000 rwxp 00000000 00:00 0
7f4d58c48000-7f9822c6f000 r-xs 00000000 e77:5ea40 144117842399223255     /app/db/gxdb/all.gxi
7f9822c6f000-7fc16bb39000 r-xs 00000000 e77:5ea40 144117842399216178     /app/db/gxdb/all.gxs
7fc16bb39000-7fc16bb3d000 r--p 00000000 00:00 0                          [vvar]
7fc16bb3d000-7fc16bb3f000 r-xp 00000000 00:00 0                          [vdso]
7ffee99d9000-7ffee99fc000 rwxp 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Artifice120 commented 2 weeks ago

I can strace the PID while it is running if that helps?

Artifice120 commented 2 weeks ago

No worries,

Ended up just using galaxy.