search

veg / hyphy

HyPhy: Hypothesis testing using Phylogenies
http://www.hyphy.org
Other
217 stars 69 forks source link

Legacy Command Line Files that do not work with 2.4 alpha #936

Closed srwis closed 5 years ago

srwis commented 5 years ago

I've decided to consolidate all of the files that are currently not working into one list so it's easier to see. Will update as I retest and as they get fixed/deleted.

Seg Faults:

Not working:

To Be Removed:

srwis commented 5 years ago

SeqAlignment.bf, SeqAlignmentProt.bf, and SeqAlignmentNuc.bf are all erroring because:

The dimension of the scoring matrix (SEQ_ALIGN_SCORE_MATRIX) was not the expected dimension: 5x5 in call to AlignSequences(aligned, inStr, alignOptions);

so it's expecting a 5x5 matrix for nucleotides (which have 4 states) and it returns the same error but expecting 21x21 for amino acids.

which has to do with this line in batchlanruntime.cpp expected_dimension = do_codon ? codon_count + 1UL : char_count + 1UL;

I think this might be already on your radar because of #863 but if not, is there a reason it's expecting a matrix with one more state than the data type has?

spond commented 5 years ago

Dear @srwis,

My cursory inspection of the code would imply that the +1UL is in error. I'll remove it in the next code commit.

Best, Sergei

srwis commented 5 years ago

NielsanYang.bf is behaving strangely. For me it hangs after the Beta model if I choose run all no matter how long I leave it running. if I choose run custom and just run Beta and the next model Beta+W it is fine and runs in under a minute. If I choose run custom and then choose 8 other models that aren't Beta it also runs fine.

However, I asked @rdvelazquez to check and see if it hung on Beta for him and he actually gets a seg fault so I'm not sure what is happening and have been unable to pinpoint exactly where the error is.

srwis commented 5 years ago

PairwiseRelativeRatio.bf

Tried fixing an error: Error: Attempted to delete a data set filter which is still being referenced by a likelihood function by adding a DeleteObject(lf); statement as suggested for a similar error in #858 but that basically allows it to run one file and then returns a seg fault:

` RUNNING SINGLE FILE ANALYSES

File /home/swisotsky/Dropbox/hyphy_tests/data/sim_BUSTED/CV1_100_replicate.1 : ln-likelihood = -1269.677204654311Segmentation fault (core dumped)`

the commit in which the seg fault is occurring can be found in #938

srwis commented 5 years ago

MEDS.bf

Would this be a candidate for the https://github.com/veg/hyphy-analyses ? It basically already has a write up at http://www.hyphy.org/w/index.php/MEDS with some associated processing scripts.

spond commented 5 years ago

Dear @srwis,

Yes, I think it would be.

Best, Sergei

spond commented 5 years ago

Dear @srwis,

PairwiseRelativeRatio.bf

Tried fixing an error: Error: Attempted to delete a data set filter which is still being referenced by a likelihood function by adding a DeleteObject(lf); statement as suggested for a similar error in #858 but that basically allows it to run one file and then returns a seg fault:

` RUNNING SINGLE FILE ANALYSES

File /home/swisotsky/Dropbox/hyphy_tests/data/sim_BUSTED/CV1_100_replicate.1 : ln-likelihood = -1269.677204654311Segmentation fault (core dumped)`

the commit in which the seg fault is occurring can be found in #938

This is a fairly obscure error that has to do with my fixing an old "fragility" in HyPhy which was not backward compatible. I'll look into it. The DeleteObject fix should work, but maybe it triggers other issues.

Best, Sergei

srwis commented 5 years ago

SubtreeSelectionComparison.bf The error is:

Error: Invalid Matrix Index [0][0] in a 0 by 7 matrix.

Function call stack 1 : doneSites[siteMap][0]=nFactor_Tree

where doneSites = {filteredData.unique_sites,7}; I think this is happening because of something to do with how the unique_sites are being counted for the filteredData. I tried 3 different files and they all gave me filteredData.unique_sites = 0. From what I can tell from looking at p51.nex there should be some unique sites when it's filtered by codons. When it is unfiltered there are unique sites see: input_p51.txt

So it's either that the three files I tried all have no unique sites when filtered by codons or what's calculating the unique sites on filtered data is wonky. from what I can tell that was site.cpp but it looks like that file has changed in the dev branch and I'm not sure what's responsible for it now. Let me know what you think. Thanks!

srwis commented 5 years ago

MFPositiveSelection.bf

So this requires a file that lists subset lengths as well as a file with trees. I haven't been able to figure out what the subset list should look like. (Is it CSV, just a bunch of numbers on new lines, do I need to list start and finish of the subsets....?) So I think this file need clearer instructions about file requirements prior to use or should be removed if it is not commonly used.

srwis commented 5 years ago

@spond I noticed while trying to resolve the issue with StarDecomposition.bf that there is an entirely separate folder for Topology inference that overlaps with a lot of the phylogeny reconstruction options.

I'd suggest either deleting that folder or moving all of the relevant bfs to it so there is less duplication. Let me know what you'd prefer.

spond commented 5 years ago

Dear @srwis,

Yes the topology inference folder had to do something with the no-longer existing GUI and should be deleted.

Best, Sergei

srwis commented 5 years ago

SubtreeSelectionComparison.bf The error is:

Error: Invalid Matrix Index [0][0] in a 0 by 7 matrix. Function call stack 1 : doneSites[siteMap][0]=nFactor_Tree

where doneSites = {filteredData.unique_sites,7}; I think this is happening because of something to do with how the unique_sites are being counted for the filteredData. I tried 3 different files and they all gave me filteredData.unique_sites = 0. From what I can tell from looking at p51.nex there should be some unique sites when it's filtered by codons. When it is unfiltered there are unique sites see: input_p51.txt

So it's either that the three files I tried all have no unique sites when filtered by codons or what's calculating the unique sites on filtered data is wonky. from what I can tell that was site.cpp but it looks like that file has changed in the dev branch and I'm not sure what's responsible for it now. Let me know what you think. Thanks!

Hey Sergei,

I think df.unique_site issue is actually impacting more bfs. I was trying to use the StripStopCodons.bf or the CleanStopCodons.bf as a basis to strip Serines for the Triple Hit model but I am running into the same issue where datafilter.unique_sites is 0. This makes me suspect that the stop codon files (or which there should probably be just one) are not actually working either. because if there are no unique sites it just says that there are no stop codons.

Here's the serine file: StripSerines.bf.txt

Best, Sadie

spond commented 5 years ago

Dear @srwis,

We should deprecate SubtreeSelectionComparison (this is effectively an obsolete version of Contrast-FEL).

In 2.4, unique_sites is only defined for DataSet objects. The way to count the number of patterns in a DataSetFilter is to look at the dimension of the pattern counts in filter.site_freqs (it will have as many rows as there are unique site patterns).

Best, Sergei

srwis commented 5 years ago

Dear @srwis,

My cursory inspection of the code would imply that the +1UL is in error. I'll remove it in the next code commit.

Best, Sergei

Hey Sergei,

Checking these I see that the +1UL was deleted but then reverted back. See the history here around April 1st/2nd https://github.com/veg/hyphy/blob/63958f0c3aba5a992e3f3f6274af7ebb97a1bab4/src/core/batchlanruntime.cpp#L940 . Curious which way is correct.

I checked and reverting to not having the +1UL solves the issue mentioned for the SeqAlign bfs not having the expected matrix dimensions but the bfs return a different error anyway. So it is not like this is the sole issue in their implementation.

Best, Sadie

srwis commented 5 years ago

I went ahead and removed all the currently none working legacy files both from the prompt and TemplateBatchFiles in #964 . These include:

I did not remove TopologySearchConstrained.bf from the TemplateBatchFiles folder as that still results in a seg fault and figured we should try and address that before the next release. I did remove it from files.lst.

I'll go through the files I did remove and leave a comment for each detailing the errors I've seen so far and how far I got along in troubleshooting as well just so we have a record of those.

srwis commented 5 years ago

DatedTipsMolecularClock.bf

This one seems very error prone or perhaps I was using bad data sets. Either way, I discover the following 3 errors when using hiv1_transmission.txt:

  1. You can't use a tree that is partitioned like branchname_date {label}. if you do you get this error: Error: Model is not associated with the node:0557_24 Function call stack 1: LikelihoodFunction lf = (filteredData, givenTree) ; My suggestion here is to have this error be more informative or state at the input step that partitioned trees will not work.

  2. Using a none labeled tree, I chose the option to have both nonsyn and syn rates locally constrained and got the following error: Parsing error (parameter list is out of context in the following context: clockTree.0564_7.synRate,<ERROR HERE>nonSynRate:=clockTree_sc’) while compiling the statement: clockTr ee.0564_7.synRate,nonSynRate:=clockTree_scaler_0*(0.6999999999999998-clockTree.Node26_1.T) Function call stack 1 : clockTree.0564_7.synRate,nonSynRate:=clockTree_scaler_0*(0.6999999999999998-clockTree.Node26_1.T); 2 : ExecuteCommands(generateDatedTipConstraints(“clockTree”,parameter2ConstrainString,tipDateAVL,0,_initialGuesses), /home/swisotsky/hyphy/res/TemplateBatchFiles/); where: parameter2ConstrainString synRate,nonSynRate So looks like it tries to constrain synRate,nonSynRate at once and the comma throws an error. My suggestion here is to either write it so it loops over parameter2ConstrainString or throw some if statements in and have it constrain each rate individually.

  3. However, even if when I choose to constrain just the synonymous rate I got the following error: Error: Failed to compute a valid transition matrix; this is usually caused by ill-conditioned rate matrices (e.g. very large rate values) Function call stack 1 : Optimize(res1, lfConstrained); This is a new to me error and I do not currently know how to handle it.

additionally, I used this data set integrase_BDA.nex.txt

as well to test and got a fourth error: Error: Internal error, dumping the offending likelihood function to /tmp/hyphy.dumpConstrained optimization failed, since a starting point within the domain specified for the variables couldn’t be found. Set it by hand, or check your constraints for compatibility. Failed constraint:clockTree.A1_KE_99_KNH1088.t:=clockTree_scaler_0[3.64174:0-10000]*(0.0248996-clockTree.Node2.T[10.2351:0-0.0205024]) must be in [0,10000]. Current value = -37.183.

Function call stack 1 : Optimize storing into, res1, the following likelihood function:lfConstrained ; 

srwis commented 5 years ago

LEISR.bf

upon retesting on 5-21 this is giving me the same error as 3) from the DatedTipsMolecularClock.bf see for full error: LEISR error.txt

srwis commented 5 years ago

FitnessAAModels.bf New error as of 5-21: Error: Internal error Likelihood function evaluation encountered a NaN (probably due to a parameterization error or a bug). Function call stack 1 : Optimize(res, lf); fitnessAA.txt

srwis commented 5 years ago

SequentialAdditon.bf New seg fault. See file for more info: sequentialaddition.txt

spond commented 5 years ago

Dear @srwis,

Please confirm that c7fa119fa35059fe52f3249c12c79679340f8212 fixes the issues. Great work finding them!

Best, Sergei

srwis commented 5 years ago

Hey @spond ,

So I tried just pulling the latest develop branch and with a fresh clone of the latest dev branch and I'm still getting the same error for LEISR. This is what I did for the fresh install: 1166 git clone https://github.com/veg/hyphy.git 1165 git remote add veg https://github.com/veg/hyphy.git 1166 cd hyphy/ 1167 git remote add veg https://github.com/veg/hyphy.git 1168 git fetch veg develop 1169 git branch 1170 git checkout --track veg/develop 1171 git pull 1172 git branch 1173 cmake . 1174 make install 1175 make -j MP

I also checked with git log and it looks like the commit is there so I'm not sure what's up: git log | grep c7fa119 Merge: 2b918c54 c7fa119f commit c7fa119fa35059fe52f3249c12c79679340f8212

Best, Sadie

spond commented 5 years ago

Dear @srwis,

I can't replicate LEISR errors with the standard test. Can you please provide the entire (input + settings) configuration that triggers the errors?

Best, Sergei

srwis commented 5 years ago

Hey @spond

here's the input: LEISR error.txt with this data set: ksr2.fna.txt

this was also on the cluster so I'm testing locally on my mac to see if I get the same error.

Locally on the mac I'm getting the same error using the p51.aa file as well. Input, along with error and message log: LESIR_input_p51.txt errors.log messages.log

Best, Sadie

spond commented 5 years ago

Dear @srwis,

Resolved the LEISR issue; there was a problem with allowing the rate parameter to become < 0 in the file (fixed). Also I found an unrelated issue that JC69 has one random entry as 10.7 (instead of 1). Should be fixed with fcbabe162ef7d434d3c518efeeed71f77fa601e9

Best, Sergei

srwis commented 5 years ago

Checked and LEISR is fixed! Also checked the rest of the recent issues I posted and all but FitnessAAModels.bf seem to run fine now. I get this error with the p51.nex data set: Error: Internal error Likelihood function evaluation encountered a NaN (probably due to a parameterization error or a bug). Function call stack 1 : Optimize(res, lf);

Here's the input for that: FitnessAA_input.txt

sdwfrost commented 5 years ago

Can I add a postprocessing failure? When fitting a model using hyphy -p, and selecting (9) for Ancestral Sequences, I get the following:

Error:
Likelihood Function/SCFGlikelihoodFunctionName__ has not been initialized in call to DataSet ancestralSequences = ReconstuctAncestors(likelihoodFunctionName__);

Function call stack
1 :  DataSet ancestralSequences = ReconstuctAncestors(likelihoodFunctionName__);