Multiple errors with the same dataset

[tshori]

So I was trying to run stringtie on a fish dataset and at first I got a segmentation fault error. I did some research, and saw that this could have been an issue with threads. So I removed the p option and I got a malloc_consolidate() invalid chuncksize error. Long story short, I came to realise that I could run the same command on the same data multiple times and have different outcomes, including full success. So I coded a little python wrapper to run the same command until a return value of 0 was obtained. I found that the same command and the same data run repetitively will generate multiple errors and those are never in the same bundle (even if the error is the same). The errors I have seem are segmentation fault, malloc_consolidate(): invalid chuncsize, input is not sorted (when the input is sorted - I checked multiple times) and a really weird onem which is an input is not sorted error that does not cause stringtie to exit. More surprisingly is that I have tried this with multiple samples (all aligned with hisat and --dta option) and the same happens with all samples, and all samples eventually will run to completion. I haven been able to see any trends. Sometimes it takes 8 runs, sometimes the first one works, but I have not had a sample that fails completely. I an using the pre-compiled binary for linux, version 2.0.5. on ubuntu 18.04. Here is one of the bam files from hisat. I am not sure what is going on, I am basically running every sample until it works and I can update on what happens downstream.

https://www.dropbox.com/s/r05ywcsedo606bx/B3_Sk1.sorted?dl=0

Here is hisat command I ran: hisat2 -q --dta --fr -p 10 -x index -1 R1.fq -2 R2.fq

I sorted the sam output with samtools using a pipe (samtools v2) and stringtie

stringtie', sortedbam, '-v', '--fr', '-B', '-o', gtfOut, '-G', genomic.gtf'

[gpertea]

Ouch, this seems to be (again!) the issue with the statically linked binary I built on CentOS 5, which somehow does not play nicely with threads (not sure why, ugh). I thought I solved it but it looks like it came back to haunt us. Sorry about that!

(Btw, even when not using the -p option, stringtie still creates a worker thread which runs in parallel with the main BAM reading/bundling code.)

Just to confirm that it's that weird linking issue that's the culprit here, is it possible to build stringtie from source on your Ubuntu system (make release) and run that instead? That seemed to have solved this stability issue in the past.. Alternatively I can build a dynamically linked version on CentOS 7, which should surely work on Ubuntu 18, and that should also (hopefully) be free of such aberrant behavior.

[tshori]

Yep. I will try that later today and get back to you! Thanks for the quick response.

[gpertea]

I'd like to use your bam file for some more testing on here on the statically linked binary, and just to make sure I have the exact same input, could you also share that genomic.gtf file you used with the -G option ?

[tshori]

Here you go:

https://www.dropbox.com/s/tdk0663ebe8pws0/GCF_002021735.1_Okis_V1_genomic.gtf?dl=0

[gpertea]

Aww, after a few tests on your data it seems that this might in fact be a different bug than what I suggested earlier.. I just got a malloc_consolidate(): invalid chunk size error on a version of the code built locally!

Still investigating this -- it seems more serious than I thought..

[tshori]

We the same file I only got a malloc_consolidate() error, but with a different file I got the weird input file not sorted error where stringtie just stay in S state instead of exiting. I not sure if it helps, but I am also running the -e option on a separate run, and it seems that with that option the number of crashes is much lower.

[gpertea]

Still trying to track this subtle multi-threading bug, but just to not waste your time with this technical issue for now, and focus on the results, you can fully disable multi-threading in stringtie by building it from source like this :

make clean
make release nothreads

The resulting stringtie binary should be then fully thread-safe (because it's thread free). I tested that a bit just now (22 runs in a row), seems stable.

[tshori]

Thanks!

[tshori]

I meant to let you know that I tested some of the samples that I managed to get processed with stringtie using ballgown and they seem to be fine.

[gpertea]

I think I also found the problem for the multi-threading crash and the fix is in github already (ad4a455), tested 31 runs with no errors so far.. Not that multi-threading helps a lot in stringtie, the nothreads version is very fast anyway, almost as fast as a multi-threaded approach for low coverage samples (!).

[tshori]

I am going to try it later today with all my samples (using 2.0.6) and get back to you.