MCMC terminated with std::bad_alloc

[janxkoci]

Hi Milane,

I was trying to run fineRADstructure on my data and got some C++ related errors that I don't understand. Here is what I did:

1. I installed fineRADstructure using conda & bioconda channel.
2. Because I called variants with freebayes instead of Stacks/pyrad, I used Stacks' populations module (v2.0) to convert my vcf to radpainter format.
3. I run RADpainter and finestructure as suggested on your website. RADpainter finished successfully, while the MCMC step of finestructure terminated with error (I used bash -x):

+ finestructure -x 100000 -y 100000 -z 1000 r123_newrbmcl100_fb_vfq20dp10gq20_2_snps.p.haps_chunks.out r123_newrbmcl100_fb_vfq20dp10gq20_2_snps.p.haps_chunks.mcmc.xml
################################################### 100%
################################################### 100%
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
fineRADstructure.sh: řádek 5: 22975 Neúspěšně ukončen (SIGABRT)        finestructure -x 100000 -y 100000 -z 1000 ${1}_chunks.out ${1}_chunks.mcmc.xml

*p.haps_chunks.out was produced by RADpainter. I tried this on two different machines with the same error:

• First I run it on a cluster with PBS system, using one node with 80 cores and 192 GB RAM. After it failed I read the website of finestructure, where they report problems with PBS, so I decided to run it on other machine.
• Second I run it on a single node without PBS, that has 64 cores and 1 TB of RAM. It was used by other users and processes, but I still had more than half of the cores and memory available. I put these specs here as I found that memory could be related to the error.

BTW my dataset has 150 samples and the input filtered VCF before processing by stacks had 400k SNPs.

Thanks for any help

Jan

[millanek]

It seems that several people get this problem with the finestructure step. Maybe five similar reports already. But it is not clear to me why this happens. Could you share with me your r123_newrbmcl100_fb_vfq20dp10gq20_2_snps.p.haps_chunks.out file, so that I can investigate it?

You can also try downloading finestructure version 4 (https://people.maths.bris.ac.uk/~madjl/finestructure/finestructure.html) and see if the updates Dan Lawson made to that help in any way... The command to issue there is “fs fs” instead of “finestructure”. Then just give it your chunks.out file.

[janxkoci]

I already tried fs v2.1.3 (it's in bioconda) but there were some extra steps needed that I didn't know how to do at the moment. I can try to give it another go.

Meanwhile, I tried to remove some outliers with lots of missing data and now I see different error (on the non-PBS node):

34809 Neoprávněný přístup do paměti (SIGSEGV)

It translates to something like "unauthorized access to memory".

Here is the first chunks.out file and here is the newer file with outliers removed.

[danjlawson]

Hi,

Try this repo:

https://github.com/danjlawson/fineRADstructure

I think there is a bug and I think it is fixed here.

Best

Dan On Jun 3, 2019, 8:12 AM +0100, janxkoci notifications@github.com, wrote:

I already tried fs v2.1.3 (it's in bioconda) but there were some extra steps needed that I didn't know how to do at the moment. I can try to give it another go. Meanwhile, I tried to remove some outliers with lots of missing data and now I see different error (on the non-PBS node): 34809 Neoprávněný přístup do paměti (SIGSEGV) It translates to something like "unauthorized access to memory". Here is the first chunks.out file and here is the newer file with outliers removed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[janxkoci]

Thanks @danjlawson

Do you have any advice on how to add GSL to my $LD_LIBRARY_PATH without root?

Your branch requires GSL 2.5, but conda has only version 2.4 (in conda-forge channel). So I tried to install GSL from source. I completed successfully the ./configure && make step but make install fails (as I don't have write permissions to /usr).

I then tried to export LD_LIBRARY_PATH=/home/koci/bin/gsl-2.5:$LD_LIBRARY_PATH but the subsequent ./configure within cloned fineRADstructure directory still fails, as the installer only sees GSL from anaconda and for some reason ignores $LD_LIBRARY_PATH. Maybe I use wrong path to GSL but I couldn't find the right subfolder that should be added to $LD_LIBRARY_PATH- any advice?

Thanks a lot again for your help.

Jan

[janxkoci]

Update

The last error message of finestructure (about unauthorized access to memory) looked suspicious so I just tried to rerun the last script with same settings and files and everything - and it finished successfully. What gives :laughing:

The 1TB node was now mostly free, only few cores were used for normal structure. I think I just got enough memory this time.

Now it got stuck on the treebuilding step throwing Lgamma error but that error has already been reported #3 and #6 .

So to summarize I think after removing outlier samples with lots of missing data, it was just a memory problem or something like that. I'm closing this issue.

I think I should repost my question about GSL 2.5 to @danjlawson fork, or is it also useful here? What you guys think?

[millanek]

Hi guys

I added Dan Lawson's putative fix for the tree building issue to my repo. Let me know if that helps.

Cheers Milan

[A33miller]

Hi millanek, I believe I am getting the same error message as was previously discussed in this thread. Can I please just clarify if the solution was to use the "bug fix" here: https://github.com/danjlawson/fineRADstructure/blob/master/finesfunctions.cpp

Thank you, Allison

"terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc /var/spool/slurm/job12843709/slurm_script: line 13: 229550 Aborted (core dumped) finestructure -x 50000 -y 100000 populations.haps_chunks.out populations.haps_chunks.mcmc.xml"

I am running fineRADstructure/0.3.2r109-GCCcore-7.4.0

[MelinaGia]

Hello! I am also getting this error, and it is with a version of fineRADstructure downloaded yesterday from bioconda, so I assume this "bug fix" is in there. Any advice on how to clear this error?

[janxkoci]

@MelinaGia Hi, looking at releases here I see there was only one release for Bioconda and it happened about 2 months before this issue has been fixed (by Dan Lawson). So no, Bioconda doesn't have this fix yet.

@millanek could you maybe create a new release with the fix, so Bioconda can use it? Thanks a ton :)

[ddols]

Hi! I guess the bug hasn't been fixed yet for bioconda because I tried to run finestructure and it kept failing the run. Any chances said changes will be fixed for Bioconda any time?

Thanks a bunch:)