std::bad_alloc error on machine with plenty of spare RAM

[iqbal-lab]

I've just tried to benchmark build for the new Plasmodium PRG at k=5, which I have previously done on cluster node, using 35Gb of RAM.

I re-ran the same command (not via LSF, directly logged into the server), on the same commit of gramtools (using a venv), on an isolated server that only I was logged into, with no other processes running, and 256GB of RAM, and this time it crashed:

Command was

source /nfs/research1/zi/software/sourceme source /nfs/research1/zi/software/gramtools_tip_commit442/bin/activate gramtools build --gram-directory results/gramk5 --vcf prg_construction/data/pf3k_and_DBPMSPS1and2.vcf --reference reference/Pfalciparum.genome.fasta --max-read-length 150 2>> error_buildk5 1>> output_buildk5

Stdout said

Finished printing linear PRG. Final number in alphabet is 3216844 maximum thread count: 1 Executing build command Generating integer encoded PRG Number of charecters in integer encoded linear PRG: 47471952 Generating FM-Index Generating PRG masks Generating kmer index

and stderr said:

2018-05-22 14:17:36,336 gramtools INFO Start process: build 2018-05-22 14:18:10,648 gramtools INFO stdout: 2018-05-22 14:18:11,421 gramtools INFO stdout: 2018-05-22 16:31:23,599 gramtools INFO Process termination message: terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc 2018-05-22 16:31:23,599 gramtools INFO Process termination code: -6 2018-05-22 16:31:23,599 gramtools ERROR Error code != 0 2018-05-22 16:31:29,096 gramtools INFO End process: build

I can give you login access to this server.

[iqbal-lab]

I reran with --debug but didn't get any more information. Takes about 2 hours to hit the bug

[ffranr]

@iqbal-lab What was the memory use at the time of the crash please?

[iqbal-lab]

no idea, this was without LSF

[iqbal-lab]

I could run with syrupy though

[iqbal-lab]

BTW - why is help needed with this? You mean because you want to knwo the RAM at time of crash or something else?

[ffranr]

@iqbal-lab Yes please, with syrupy if possible. I would also like to know which version of gramtools was used (commit hash).

There's something very odd at work if the infrastructure changes you've described are the only varying parameters. In it's current state, gramtools build should be very simple for k = 5.

[iqbal-lab]

Commit 442ddfe7d68fca86ad97a123792673edff815b04 I'll rerun when I can with syrupy, bit hectic. Might need to ask you to do it as I go to france next week and Peru the following- have given you the command, the data, the server and the commit.

[ffranr]

At commit 442ddfe7d68fca86ad97a123792673edff815b04 gramtools is in the following state:

• If the kmer size is 8 or less: all possible kmers of that size are generated whilst building the kmer index. The PRG is not examined. (see: https://github.com/iqbal-lab-org/gramtools/commit/ff3e2d6f1b138c6839d8db7d83b51df97ef1e3a7)
• The default kmer size is 15.

The build command that you've quoted above does not specify a kmer size. Therefore gramtools uses the default kmer size of 15. The default kmer size was reduced to 5 in a later commit: https://github.com/iqbal-lab-org/gramtools/commit/5f1222303a8d1512dbd257c5a643924c0b0df552

We've already seen that a kmer size of 15 can cause build to run out of resources when processing a complicated graph.

@iqbal-lab The solution in this case is to use the latest gramtools commit.

[iqbal-lab]

Doh. OK will rerun

[iqbal-lab]

Confirmed the bug is gone at tip. I'll close this