Core dump in hmmufotu-build

Grice-Lab / HmmUFOtu

An HMM and Phylogenetic Placement based Ultra-Fast Taxonomy Assignment Tool for 16S sequencing

GNU General Public License v3.0

10 stars 9 forks source link

Core dump in hmmufotu-build #6

Open jshleap opened 5 years ago

jshleap commented 5 years ago

Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignments:

MSA loaded
MSA pruned
MSA database created for 9742 X 242 aligned sequences
CSFM index built
hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed.
Aborted (core dumped)

Any ideas on how I can overcome it?

Best,

Sergio

e00011027 commented 5 years ago

Hi Sergio,

This looks like a out of RAM error, if you are running it under bsub or ibash on a cluster. Try to allocate my RAM and SWAP before running it. If it is not, can you send me your input files to take a look?

Best Qi

On Tue, Sep 3, 2019 at 8:27 PM Jose Sergio Hleap notifications@github.com wrote:

Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignment:

MSA loaded MSA pruned MSA database created for 9742 X 242 aligned sequences CSFM index built hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped)

Any ideas on how I can overcome it?

Best,

Sergio

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Grice-Lab/HmmUFOtu/issues/6?email_source=notifications&email_token=ABXZ44OBCVWAGSDKY7RHO6LQH36HBA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJDXY6A, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXZ44JPDT2ZW3U7DP6YVJ3QH36HBANCNFSM4ITMBYYQ .

jshleap commented 4 years ago

Hi Qi, It is still not working, even with 128Gb of RAM... Do you have an idea of the required RAM based on the size of the file? None of the files I am trying to run are bigger than 200M. Where can I send you the data? Do you still have the zhengqi@pennmedicine.upenn.edu?

jshleap commented 4 years ago

Hey! I tried with 502GB of ram and had the same error:

MSA loaded
MSA pruned
MSA database created for 16657 X 313 aligned sequences
CSFM index built
hmmufotu-build: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/eigen/3.3.5/include/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed.
/var/spool/slurmd/job21271023/slurm_script: line 8: 27484 Aborted                 (core dumped)

I used the following command: hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.tree --fmt 'fasta' -V -v -p 32

MIS_realized.reference fasta en tree being the fasta-formated sequence file and the newick-formatted tree, respectively.

e00011027 commented 4 years ago

For now this looks like an error caused by all zero branch lengths of some internal node of your newick tree. Try to parse/check your customized tree to make sure at least one non-zero branch exists for every node, otherwise the matrix operation calculating tree likelihood will fail due to numerical error. It is even better if ALL branch lengths are greater than zero.

Let me know whether it works.

Best Qi

On Tue, Oct 29, 2019, 6:07 PM Jose Sergio Hleap notifications@github.com wrote:

Hey! I tried with 502GB of ram and had the same error:

MSA loaded MSA pruned MSA database created for 16657 X 313 aligned sequences CSFM index built hmmufotu-build: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/eigen/3.3.5/include/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. /var/spool/slurmd/job21271023/slurm_script: line 8: 27484 Aborted (core dumped)

I used the following command: hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.tree --fmt 'fasta' -V -v -p 32

MIS_realized.reference fasta en tree being the fasta-formated sequence file and the newick-formatted tree, respectively.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Grice-Lab/HmmUFOtu/issues/6?email_source=notifications&email_token=ABXZ44N7YCOXVVS6AEOJPBLQRCX2NA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECSH35Q#issuecomment-547651062, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXZ44MJV2KWXWSRCHBWHJ3QRCX2NANCNFSM4ITMBYYQ .

jshleap commented 4 years ago

Hi Qi! It does not work. I added 1E-10 to every branch length in the tree and it does not work. Please let me know how to send you the alignment and the tree (too big for github). Here is the command launched and its error:

hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.hmmufotu.tree --fmt 'fasta' -p 20

hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped)

e00011027 commented 4 years ago

Hi, you can try to share me a link for the files via Google Drive, Drop Box or something similar.

Best Qi

On Wed, Oct 30, 2019 at 2:42 PM Jose Sergio Hleap notifications@github.com wrote:

Hi Qi! It does not work. I added 1E-10 to every branch length in the tree and it does not work. Please let me know how to send you the alignment and the tree (too big for github). Here is the command launched and its error:

hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.hmmufotu.tree --fmt 'fasta' -p 20

hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Grice-Lab/HmmUFOtu/issues/6?email_source=notifications&email_token=ABXZ44JEZZIGKN3V2LNYRSLQRHIRNA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECVLFIY#issuecomment-548057763, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABXZ44MJSFS6WGBBKZAVHNTQRHIRNANCNFSM4ITMBYYQ .

jshleap commented 4 years ago

Hi Qi, Here is the link: https://drive.google.com/file/d/1sSPjjQX3PKUA28BY3hXMxwq8nQRgCGtM/view?usp=sharing Best

e00011027 commented 4 years ago

Hi Sergio,

After testing we found it is caused by a bug for certain MSA profiles with all sites consensus. The GitHub source code has been updated and you should pull the latest to remake HmmUFOtu.

However, after fixing the bug your input still won't build into a database, because the "Unmatched MSA and Tree. Found 16657 leaf sequences from MSA but expecting 27664 leaves in the Phylogenetic Tree". In another word, your tree file contains more leaf nodes than the sequences listed in the .fasta file. If it is just because you truncated the input for easier transfer than ignore it. Otherwise you need to re-build your tree only based on these aligned sequences. Please let us know if you have any questions.

Best Qi

On Tue, Sep 3, 2019 at 8:27 PM Jose Sergio Hleap notifications@github.com wrote:

Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignment:

MSA loaded MSA pruned MSA database created for 9742 X 242 aligned sequences CSFM index built hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped)

Any ideas on how I can overcome it?

Best,

Sergio

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Grice-Lab/HmmUFOtu/issues/6?email_source=notifications&email_token=ABXZ44OBCVWAGSDKY7RHO6LQH36HBA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJDXY6A, or mute the thread https://github.com/notifications/unsubscribe-auth/ABXZ44JPDT2ZW3U7DP6YVJ3QH36HBANCNFSM4ITMBYYQ .