search

ComparativeGenomicsToolkit / cactus

Official home of genome aligner based upon notion of Cactus graphs
Other
523 stars 111 forks source link

boost some job memory specs in light of 16-fly logging #1141

Closed glennhickey closed 1 year ago

glennhickey commented 1 year ago

Rerunning the 16-fly pangenome on slurm failed completely due to some memory underestimates. This PR makes some adjustments based on the following logging information

grep WARNING 16-fly-pg-2023-08-23.stderr
[2023-08-23T13:32:47-0700] [MainThread] [I] [toil-rt] 2023-08-23 13:32:47.958235: Successfully ran [minigraph]: "bash -c 'set -eo pipefail && minigraph -c -xggs -t 64 dm6.fa AB8.0.fa B7.0.fa A3.0.fa B6.0.fa OreR.0.fa A5.0.fa A1.0.fa B2.0.fa B4.0.fa A4.0.fa A6.0.fa B3.0.fa A2.0.fa B1.0.fa A7.0.fa | bgzip --threads 64'" in 18105.9872 seconds and 16.4 Gi memory with job-memory 9.3 Gi. Percent utilization: 177.2 **WARNING: limit exceeded**
[2023-08-23T14:06:21-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:06:21.446413: Successfully ran: "bash -c 'set -eo pipefail && minigraph /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/5f75/33bb/tmp44v749el/mg.gfa /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/5f75/33bb/tmp44v749el/A4.0.fa -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/5f75/33bb/tmp44v749el/A4.0.gaf -c -xasm -t 8'" in 1975.6473 seconds and 9.7 Gi memory with job-memory 8.7 Gi. Percent utilization: 111.4 **WARNING: limit exceeded**
[2023-08-23T14:09:37-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:09:37.365430: Successfully ran: "bash -c 'set -eo pipefail && minigraph /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/422d/6541/tmpw8p5507z/mg.gfa /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/422d/6541/tmpw8p5507z/B6.0.fa -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/422d/6541/tmpw8p5507z/B6.0.gaf -c -xasm -t 8'" in 2171.5689 seconds and 8.5 Gi memory with job-memory 8.5 Gi. Percent utilization: 100.3 **WARNING: limit exceeded**
[2023-08-23T14:12:13-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:12:13.662030: Successfully ran: "bash -c 'set -eo pipefail && minigraph /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/1555/c33f/tmpib8td66d/mg.gfa /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/1555/c33f/tmpib8td66d/A5.0.fa -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/1555/c33f/tmpib8td66d/A5.0.gaf -c -xasm -t 8'" in 2327.8609 seconds and 8.7 Gi memory with job-memory 8.6 Gi. Percent utilization: 101.2 **WARNING: limit exceeded**
[2023-08-23T14:22:35-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:22:35.713389: Successfully ran: "bash -c 'set -eo pipefail && minigraph /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/90c0/0335/tmpd8gih3zt/mg.gfa /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/90c0/0335/tmpd8gih3zt/B7.0.fa -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/90c0/0335/tmpd8gih3zt/B7.0.gaf -c -xasm -t 8'" in 2949.8948 seconds and 9.7 Gi memory with job-memory 8.8 Gi. Percent utilization: 110.4 **WARNING: limit exceeded**
[2023-08-23T14:24:01-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:24:01.691273: Successfully ran: "bash -c 'set -eo pipefail && minigraph /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/9ab7/3f68/tmpp_d3zq02/mg.gfa /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/9ab7/3f68/tmpp_d3zq02/AB8.0.fa -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/9ab7/3f68/tmpp_d3zq02/AB8.0.gaf -c -xasm -t 8'" in 3035.9154 seconds and 8.5 Gi memory with job-memory 8.5 Gi. Percent utilization: 100.0 **WARNING: limit exceeded**
[2023-08-23T14:39:21-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:39:21.296414: Successfully ran: "halAppendCactusSubtree /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/f3c8/d520/tmpxwuih3p8/Anc0.hal.c2h /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/f3c8/d520/tmpxwuih3p8/Anc0.hal.fa '(dm6:0.025,AB8.0:0.025,B1.0:0.025,B2.0:0.025,B3.0:0.025,B6.0:0.025,OreR.0:0.025,_MINIGRAPH_:0.025)Anc0;' /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/f3c8/d520/tmpxwuih3p8/Anc0.hal --inMemory" in 0.1515 seconds and 9.3 Mi memory with job-memory 2.2 Mi. Percent utilization: 421.6 **WARNING: limit exceeded**
[2023-08-23T14:39:21-0700] [MainThread] [I] [toil-rt] 2023-08-23 14:39:21.485931: Successfully ran: "halRemoveDupes /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/f3c8/d520/tmpxwuih3p8/Anc0.hal dm6" in 0.081 seconds and 6.2 Mi memory with job-memory 2.2 Mi. Percent utilization: 282.0 **WARNING: limit exceeded**
[2023-08-23T15:09:33-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:09:33.335468: Successfully ran: "odgi sort -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/cbbe/0bdb/tmp9z832vxy/chrY.full.og -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/cbbe/0bdb/tmp9z832vxy/chrY.full.og.sort -t 32" in 0.2138 seconds and 16.7 Mi memory with job-memory 8.6 Mi. Percent utilization: 195.6 **WARNING: limit exceeded**
[2023-08-23T15:09:33-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:09:33.615510: Successfully ran: "odgi viz -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/cbbe/0bdb/tmp9z832vxy/chrY.full.og.sort -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/cbbe/0bdb/tmp9z832vxy/chrY.full.og.viz.png -M /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/cbbe/0bdb/tmp9z832vxy/path_sample_names -t 32 -x 1500 -y 500 -a 10" in 0.2371 seconds and 19.9 Mi memory with job-memory 8.6 Mi. Percent utilization: 232.7 **WARNING: limit exceeded**
[2023-08-23T15:09:35-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:09:35.406946: Successfully ran: "odgi sort -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/0c0e/8962/tmp2eweni0a/chrM.full.og -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/0c0e/8962/tmp2eweni0a/chrM.full.og.sort -t 32" in 0.082 seconds and 12.4 Mi memory with job-memory 588.3 Ki. Percent utilization: 2.153e+03 **WARNING: limit exceeded**
[2023-08-23T15:09:35-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:09:35.766689: Successfully ran: "odgi viz -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/0c0e/8962/tmp2eweni0a/chrM.full.og.sort -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/0c0e/8962/tmp2eweni0a/chrM.full.og.viz.png -M /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/0c0e/8962/tmp2eweni0a/path_sample_names -t 32 -x 1500 -y 500 -a 10" in 0.3269 seconds and 16.7 Mi memory with job-memory 588.3 Ki. Percent utilization: 2.905e+03 **WARNING: limit exceeded**
[2023-08-23T15:10:23-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:10:23.837286: Successfully ran: "odgi sort -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/404f/2528/tmp835_7_ub/chr4.full.og -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/404f/2528/tmp835_7_ub/chr4.full.og.sort -t 32" in 16.5995 seconds and 78.6 Mi memory with job-memory 67.5 Mi. Percent utilization: 116.5 **WARNING: limit exceeded**
[2023-08-23T15:10:27-0700] [MainThread] [I] [toil-rt] 2023-08-23 15:10:27.481167: Successfully ran: "odgi viz -i /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/404f/2528/tmp835_7_ub/chr4.full.og.sort -o /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/404f/2528/tmp835_7_ub/chr4.full.og.viz.png -M /data/tmp/714c710ba4dc5bc0b7a52aa43b37d31a/404f/2528/tmp835_7_ub/path_sample_names -t 32 -x 1500 -y 500 -a 10" in 2.8518 seconds and 76.2 Mi memory with job-memory 67.5 Mi. Percent utilization: 113.0 **WARNING: limit exceeded**