gwct
commented
5 years ago Hi, I'm not sure I have a solution for you right off the bat, but I just wanted to note that an error rate of 0.8125 is extremely high. That basically means that 81% of your gene counts are wrong. I usually observe error rates in the range of 0.01 to 0.15. So, while this doesn't address the error directly it may point to a problem in the underlying data. If you're confident in your data, then I would suggest running CAFE without the error model. caferror has already done this in the first run and it looks like it completed successfully:
CAFE run number 1 Pre run (no error models)
No error models generated
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 178448.618943 Lambda with above error models: 0.00704727021613
Hopefully that helps a bit. -Gregg
On Thu, Aug 22, 2019 at 1:38 PM Kristina Gagalova notifications@github.com wrote:
Hi, I have some awkward results after running cafe with the error model correction. I have followed the tutorial proposed here http://evomicsorg.wpengine.netdna-cdn.com/wp-content/uploads/2016/06/cafe_tutorial-1.pdf .
The error comes after the error correction. This is the error log file
=========================================================================
Assembly/annotation error estimation
07.25.2019 | 04:46:09
Using CAFE shell file: shell_wholeTree_cafe1.sh
--------------------------
INPUT INFO
CAFE path set as: #!/home/kgagalova/src/CAFE/release/cafe
Using gene family file: Orthogroups.GeneCountFilt1.tsv
Using tree command: tree ((plambertiana:84.989322,ptaeda:84.989322):55.010678,(((WS77111:17.358173,PG29:17.358173):5.626297,Q903:22.984470):16.691297,pabies:39.675767):100.324233);
Using lambda command lambda -s
--------------------------
OPTIONS INFO
-c 0 Global grid search option.
-e 0.4 Starting error estimate of 0.4.
-f 1 Initial CAFE run option.
-v 0 CAFE verbosity option.
-s 0 Individual species minimization option.
--------------------------
OUTPUT INFO
Putting all files in: reports/run3_caferror_files/
Log file for all runs: reports/run3_caferror_files/caferrorLog.txt
Output file: reports/run3_caferror_files/caferror_default_output.txt
=========================================================================
Beginning Global Error Prediction...
CAFE run number 1 Pre run (no error models)
No error models generated
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 178448.618943 Lambda with above error models: 0.00704727021613
CAFE run number 2 0.4 Error Model Run for all
Generating 0.4 Error File for ptaeda
0.4 Error File Already Created for plambertiana
0.4 Error File Already Created for WS77111
0.4 Error File Already Created for pabies
0.4 Error File Already Created for PG29
0.4 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170844.974435 Lambda with above error models: 0.00638433008738
CAFE run number 3 0.2 Error Model Run for all
Generating 0.2 Error File for ptaeda
0.2 Error File Already Created for plambertiana
0.2 Error File Already Created for WS77111
0.2 Error File Already Created for pabies
0.2 Error File Already Created for PG29
0.2 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 171629.613978 Lambda with above error models: 0.00622590671489
CAFE run number 4 0.8 Error Model Run for all
Generating 0.8 Error File for ptaeda
0.8 Error File Already Created for plambertiana
0.8 Error File Already Created for WS77111
0.8 Error File Already Created for pabies
0.8 Error File Already Created for PG29
0.8 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170696.250250 Lambda with above error models: 0.00999999999882
CAFE run number 5 0.9 Error Model Run for all
Generating 0.9 Error File for ptaeda
0.9 Error File Already Created for plambertiana
0.9 Error File Already Created for WS77111
0.9 Error File Already Created for pabies
0.9 Error File Already Created for PG29
0.9 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170845.379773 Lambda with above error models: 0.00999999999988
CAFE run number 6 0.65 Error Model Run for all
Generating 0.65 Error File for ptaeda
0.65 Error File Already Created for plambertiana
0.65 Error File Already Created for WS77111
0.65 Error File Already Created for pabies
0.65 Error File Already Created for PG29
0.65 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 171239.629602 Lambda with above error models: 0.00989898874295
CAFE run number 7 0.775 Error Model Run for all
Generating 0.775 Error File for ptaeda
0.775 Error File Already Created for plambertiana
0.775 Error File Already Created for WS77111
0.775 Error File Already Created for pabies
0.775 Error File Already Created for PG29
0.775 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170736.018075 Lambda with above error models: 0.00999999999995
CAFE run number 8 0.8375 Error Model Run for all
Generating 0.8375 Error File for ptaeda
0.8375 Error File Already Created for plambertiana
0.8375 Error File Already Created for WS77111
0.8375 Error File Already Created for pabies
0.8375 Error File Already Created for PG29
0.8375 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170690.633625 Lambda with above error models: 0.00999999999793
CAFE run number 9 0.86875 Error Model Run for all
Generating 0.86875 Error File for ptaeda
0.86875 Error File Already Created for plambertiana
0.86875 Error File Already Created for WS77111
0.86875 Error File Already Created for pabies
0.86875 Error File Already Created for PG29
0.86875 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170740.814494 Lambda with above error models: 0.00999999999873
CAFE run number 10 0.41875 Error Model Run for all
Generating 0.41875 Error File for ptaeda
0.41875 Error File Already Created for plambertiana
0.41875 Error File Already Created for WS77111
0.41875 Error File Already Created for pabies
0.41875 Error File Already Created for PG29
0.41875 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170914.807286 Lambda with above error models: 0.00647607274535
CAFE run number 11 0.64375 Error Model Run for all
Generating 0.64375 Error File for ptaeda
0.64375 Error File Already Created for plambertiana
0.64375 Error File Already Created for WS77111
0.64375 Error File Already Created for pabies
0.64375 Error File Already Created for PG29
0.64375 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 171270.419272 Lambda with above error models: 0.00978977422897
CAFE run number 12 0.75625 Error Model Run for all
Generating 0.75625 Error File for ptaeda
0.75625 Error File Already Created for plambertiana
0.75625 Error File Already Created for WS77111
0.75625 Error File Already Created for pabies
0.75625 Error File Already Created for PG29
0.75625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170782.585911 Lambda with above error models: 0.00999999999901
CAFE run number 13 0.8125 Error Model Run for all
Generating 0.8125 Error File for ptaeda
0.8125 Error File Already Created for plambertiana
0.8125 Error File Already Created for WS77111
0.8125 Error File Already Created for pabies
0.8125 Error File Already Created for PG29
0.8125 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170686.883869 Lambda with above error models: 0.00999999999823
CAFE run number 14 0.840625 Error Model Run for all
Generating 0.840625 Error File for ptaeda
0.840625 Error File Already Created for plambertiana
0.840625 Error File Already Created for WS77111
0.840625 Error File Already Created for pabies
0.840625 Error File Already Created for PG29
0.840625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170693.343050 Lambda with above error models: 0.00999999999925
CAFE run number 15 0.40625 Error Model Run for all
Generating 0.40625 Error File for ptaeda
0.40625 Error File Already Created for plambertiana
0.40625 Error File Already Created for WS77111
0.40625 Error File Already Created for pabies
0.40625 Error File Already Created for PG29
0.40625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170866.770367 Lambda with above error models: 0.00641164403997
CAFE run number 16 0.6234375 Error Model Run for all
Generating 0.6234375 Error File for ptaeda
0.6234375 Error File Already Created for plambertiana
0.6234375 Error File Already Created for WS77111
0.6234375 Error File Already Created for pabies
0.6234375 Error File Already Created for PG29
0.6234375 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 171359.249841 Lambda with above error models: 0.00939990623284
CAFE run number 17 0.73203125 Error Model Run for all
Generating 0.73203125 Error File for ptaeda
0.73203125 Error File Already Created for plambertiana
0.73203125 Error File Already Created for WS77111
0.73203125 Error File Already Created for pabies
0.73203125 Error File Already Created for PG29
0.73203125 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170861.558931 Lambda with above error models: 0.00999999999556
CAFE run number 18 0.786328125 Error Model Run for all
Generating 0.786328125 Error File for ptaeda
0.786328125 Error File Already Created for plambertiana
0.786328125 Error File Already Created for WS77111
0.786328125 Error File Already Created for pabies
0.786328125 Error File Already Created for PG29
0.786328125 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170714.652895 Lambda with above error models: 0.00999999999994
CAFE run number 19 0.8134765625 Error Model Run for all
Generating 0.8134765625 Error File for ptaeda
0.8134765625 Error File Already Created for plambertiana
0.8134765625 Error File Already Created for WS77111
0.8134765625 Error File Already Created for pabies
0.8134765625 Error File Already Created for PG29
0.8134765625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170686.443001 Lambda with above error models: 0.00999999999834
CAFE run number 20 0.82705078125 Error Model Run for all
Generating 0.82705078125 Error File for ptaeda
0.82705078125 Error File Already Created for plambertiana
0.82705078125 Error File Already Created for WS77111
0.82705078125 Error File Already Created for pabies
0.82705078125 Error File Already Created for PG29
0.82705078125 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170685.292502 Lambda with above error models: 0.00999999999870
CAFE run number 21 0.833837890625 Error Model Run for all
Generating 0.833837890625 Error File for ptaeda
0.833837890625 Error File Already Created for plambertiana
0.833837890625 Error File Already Created for WS77111
0.833837890625 Error File Already Created for pabies
0.833837890625 Error File Already Created for PG29
0.833837890625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170688.158502 Lambda with above error models: 0.00999999999854
CAFE run number 22 0.413525390625 Error Model Run for all
Generating 0.413525390625 Error File for ptaeda
0.413525390625 Error File Already Created for plambertiana
0.413525390625 Error File Already Created for WS77111
0.413525390625 Error File Already Created for pabies
0.413525390625 Error File Already Created for PG29
0.413525390625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170893.987929 Lambda with above error models: 0.00644838637450
CAFE run number 23 0.623681640625 Error Model Run for all
Generating 0.623681640625 Error File for ptaeda
0.623681640625 Error File Already Created for plambertiana
0.623681640625 Error File Already Created for WS77111
0.623681640625 Error File Already Created for pabies
0.623681640625 Error File Already Created for PG29
0.623681640625 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 171358.296970 Lambda with above error models: 0.00940409362910
CAFE run number 24(FINAL RUN) With minimized error models
0.82705078125 Error File Already Created for ptaeda
0.82705078125 Error File Already Created for plambertiana
0.82705078125 Error File Already Created for WS77111
0.82705078125 Error File Already Created for pabies
0.82705078125 Error File Already Created for PG29
0.82705078125 Error File Already Created for Q903
Rewriting CAFE shell script... Running CAFE [silently] with error models listed above... Score with above error models: 170685.278231 Lambda with above error models: 0.00999999999788
=======================================================================
****
Score with no errormodel: 178448.618943
Lambda with no errormodel: 0.00704727021613
****
Global Error Estimation: 0.82705078125
Score with global errormodel: 170685.292502
Lambda with global errormodel: 0.00999999999870
****
=======================================================================
Caferror finished at: 07.25.2019 | 05:06:21
Runtime: 20.192778484 minutes
Based on the lowest score I have selected the error 0.8125 model
ErrorModel Score 0.4 178448.618943 0.4 170844.974435 0.2 171629.613978 0.8 170696.250250 0.9 170845.379773 0.65 171239.629602 0.775 170736.018075 0.8375 170690.633625 0.86875 170740.814494 0.41875 170914.807286 0.64375 171270.419272 0.75625 170782.585911 0.8125 170686.883869 0.840625 170693.343050 0.40625 170866.770367 0.6234375 171359.249841 0.73203125 170861.558931 0.786328125 170714.652895 0.8134765625 170686.443001 0.82705078125 170685.292502 0.833837890625 170688.158502 0.413525390625 170893.987929 0.623681640625 171358.296970
I have than re-run cafe with the error model but I am getting a warning and the calculated lambda is higher than the original one.
Error correction script
load -i Orthogroups.GeneCountFilt2.tsv -t 4 -l reports/log_run3error.txt tree ((plambertiana:84.989322,ptaeda:84.989322):55.010678,(((WS77111:17.358173,PG29:17.358173):5.626297,Q903:22.984470):16.691297,pabies:39.675767):100.324233); errormodel -model reports/run3_caferror_files/cafe_errormodel_0.8125.txt -all lambda -s report reports/report_run3Error
Log file
cafe shell_wholeTreeErrorCorr_cafe3.sh
Family information: Orthogroups.GeneCountFilt2.tsv Log: reports/log_run3error.txt The number of families is 19234 Root Family size : 1 ~ 111 Family size : 0 ~ 139 P-value: 0.01 Num of Threads: 4 Num of Random: 1000 ((plambertiana:84.9893,ptaeda:84.9893):55.0107,(((WS77111:17.3582,PG29:17.3582):5.6263,Q903:22.9845):16.6913,pabies:39.6758):100.324) errormodel: reports/run3_caferror_files/cafe_errormodel_0.8125.txt set. errormodel: Remember that the rows in the errormodel file have to add up to 1 (rows in the errormodel file correspond to columns in the errormatrix). errormodel: The program does not check, only renormalizes. Empirical Prior Estimation Result: (34 iterations) Poisson lambda: 0.877435 & Score: 128700.086499 Lambda : 0.00250736789778 & Score: -184547.133462 .Lambda : 0.00263273629267 & Score: -183764.447882 .Lambda : 0.00275810468756 & Score: -183033.523101 .Lambda : 0.00288347308245 & Score: -182350.154862 .Lambda : 0.00313420987222 & Score: -181108.356003 .Lambda : 0.00338494666200 & Score: -180009.888133 .Lambda : 0.00388642024156 & Score: -178158.450554 .Lambda : 0.00438789382111 & Score: -176661.464238 .Lambda : 0.00539084098022 & Score: -174440.879490 .Lambda : 0.00639378813934 & Score: -172941.192402 .Lambda : 0.00839968245756 & Score: -171248.581618 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01040557677578 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01040557677578 & Score: -inf .Lambda : 0.00940262961667 & Score: -170812.285719 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01040557677578 & Score: -inf .Lambda : 0.00990410319623 & Score: -170664.649499 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01040557677578 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01015483998600 & Score: -inf .Lambda : 0.00965336640645 & Score: -170732.847412 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01015483998600 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01002947159112 & Score: -inf .Lambda : 0.00977873480134 & Score: -170697.374863 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01002947159112 & Score: -inf .Lambda : 0.00996678739367 & Score: -170649.110067 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01002947159112 & Score: -inf .Lambda : 0.00999812949239 & Score: -170641.515088 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01002947159112 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01001380054175 & Score: -inf .Lambda : 0.00998245844303 & Score: -170645.301011 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01001380054175 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000596501707 & Score: -inf .Lambda : 0.00999029396771 & Score: -170643.402922 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000596501707 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000204725473 & Score: -inf .Lambda : 0.00999421173005 & Score: -170642.457767 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000204725473 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .Lambda : 0.00999617061122 & Score: -170641.986094 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .Lambda : 0.00999910893298 & Score: -170641.279834 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .Lambda : 0.00999959865327 & Score: -170641.162269 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .Lambda : 0.00999984351342 & Score: -170641.103494 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .Lambda : 0.00999996594349 & Score: -170641.074108 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000008837356 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000002715853 & Score: -inf .Lambda : 0.00999990472845 & Score: -170641.088801 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000002715853 & Score: -inf .Lambda : 0.00999999655101 & Score: -170641.066760 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000002715853 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000001185477 & Score: -inf .Lambda : 0.00999998124725 & Score: -170641.070435 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000001185477 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000420289 & Score: -inf .Lambda : 0.00999998889913 & Score: -170641.068598 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000420289 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000037695 & Score: -inf .Lambda : 0.00999999272507 & Score: -170641.067679 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000037695 & Score: -inf .Lambda : 0.00999999846398 & Score: -170641.066301 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000037695 & Score: -inf .Lambda : 0.00999999942046 & Score: -170641.066071 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000037695 & Score: -inf .Lambda : 0.00999999989871 & Score: -170641.065957 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000037695 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000013783 & Score: -inf .Lambda : 0.00999999965958 & Score: -170641.066014 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000013783 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000001827 & Score: -inf .Lambda : 0.00999999977914 & Score: -170641.065985 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000001827 & Score: -inf .Lambda : 0.00999999995849 & Score: -170641.065942 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000001827 & Score: -inf .Lambda : 0.00999999998838 & Score: -170641.065935 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000001827 & Score: -inf .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000000332 & Score: -inf .Lambda : 0.00999999997343 & Score: -170641.065939 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000000332 & Score: -inf .Lambda : 0.00999999999585 & Score: -170641.065933 .WARNING: Calculated posterior probability for family OG0000056 = 0 Lambda : 0.01000000000332 & Score: -inf .Lambda : 0.00999999999958 & Score: -170641.065932 .. Lambda Search Result: 34 Lambda : 0.00999999999958 & Score: 170641.065932 DONE: Lambda Search or setting, for command: lambda -s Running Viterbi algorithm.... Report Done
I have checked the gene family OG0000056 and those are the gene family counts (very first gene family):
Desc Family ID PG29 Q903 WS77111 pabies plambertiana ptaeda (null) OG0000056 10 10 14 42 8 14
Do you have any suggestion where I can start looking to fix the error? Than k you in advance
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/hahnlab/CAFE/issues/66?email_source=notifications&email_token=AC7RJCLWGC35BM4W6XUIRTDQF3TKBA5CNFSM4IOYQP72YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HG35M6Q, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7RJCP67JTYTMVDFFSZBKTQF3TKBANCNFSM4IOYQP7Q .
KristinaGagalova
Hi, I have some awkward results after running cafe with the error model correction. I have followed the tutorial proposed here.
The error comes after the error correction. Log file of the cafeerror.py
Based on the lowest score I have selected the error 0.8125 model
I have than run cafe with the error model but I am getting a warning and the error-corrected lambda is higher than the original one.
CAFE with error-correction script
Log file
I have checked the gene family OG0000056 and those are the gene family counts (very first gene family):
Do you have any suggestion where I can start looking to fix the error? Than k you in advance