reprocess_rnac.pl - Out of memory

[SuhasSrinivasan]

Dear Researchers,

Thank you for making this amazing method/modeling library available to the community.

I am currently going through the installation steps and there is a blocking issue at the reprocess_rnac.pl step.

The installation environment is a server with 32 cores and 256 GB of memory.

The Out of memory issue happens in about a minute of starting the process.

$ ../input_prep/reprocess_rnac.pl id_mapping.tsv.gz rfam_annotations.tsv.gz
Out of memory!

I was monitoring through htop and saw the process had used ~16GB of Memory but no change in Swap.

Would appreciate your input about this.

[SuhasSrinivasan]

This seems like an issue with the amount of Physical and Virtual memory available to the process, as described in ulimit.

With the below values the process would fail.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 386865
max locked memory       (kbytes, -l) 16777216
max memory size         (kbytes, -m) unlimited
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) 18874368
file locks                      (-x) unlimited

Running this on a different server with the below values resulted in completion with changes made to rfam_annotations.tsv.gz.

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1028693
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 33554432
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16384
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Closing this issue for now.