search

bowmanjeffs / paprica

paprica - PAthway PRediction by phylogenetIC plAcement
27 stars 8 forks source link

Benchmarking against PICRUst/Tax4fun? #51

Closed GeoMicroSoares closed 7 years ago

GeoMicroSoares commented 7 years ago

Hey @bowmanjeffs, congratulations for the software, looks really nice and user friendly!

Have tried testing it against other related software such as PICRUst or Tax4Fun?

Cheers, André

bowmanjeffs commented 7 years ago

André, I haven't done this extensively. I'd like to, but haven't find the right mechanism. Paprica doesn't use KEGG which makes a direct comparison challenging. Comparing predictions to metagenomes doesn't work at all for reasons explained here. I did at one point look at a random selection of draft genomes (Tax4Fun can't handle a single OTU input so doesn't work here). See figure below comparing Kendall's tau between predictions and true genome. Picrust did slightly better with most of the draft genomes but the difference wasn't great. There were however, a large number of draft genomes that Picrust couldn't predict at all because they didn't slot into the closed OTUs. Paprica was able to provide a prediction for these. A more complete comparison is definitely desirable, but I haven't had the opportunity to pursue further (along with putting out an updated methods paper!).

fig_3.pdf