RachelKarchin
commented
3 years ago Thanks for looking into this! We will take a careful look and get back to you as soon as possible.
Best
- RK
On Thu, Dec 3, 2020 at 10:49 AM vladimirkovacevic notifications@github.com wrote:
I was very curious to test the performance of mhcnuggets 2.3 on the latest dataset with TESLA validated neoantigen candidates published in Cell (Key Parameters of Tumor Epitope Immunogenicity Revealed Through a Consortium Approach Improve Neoantigen Prediction https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cell.com%2Fcell%2Ffulltext%2FS0092-8674(20)31156-9&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C0%7C637426073952070568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=cDYTIQ8nbMYc3W8eS56rjPoCUmyYdW%2B9l4ooOPSx7s4%3D&reserved=0, Table S4 https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fars.els-cdn.com%2Fcontent%2Fimage%2F1-s2.0-S0092867420311569-mmc4.xlsx&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952070568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4L3l4GDz3S4%2BRYfwUFTzR08Unh5Hi1VwEJ5YuHPC5T8%3D&reserved=0 and S7 https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fars.els-cdn.com%2Fcontent%2Fimage%2F1-s2.0-S0092867420311569-mmc7.xlsx&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952080565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DbE108rOp3TvDjpdttcJYuliqvMR0Hkm4tbwgvfDp3s%3D&reserved=0). In TESLA they used flow cytometry and microscopy to confirm (VALIDATE) neoantigen candidates. More info is available in the paper. Here are ic50 scores obtained by mhcnuggets. [image: mhcnuggets_validated] https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F16796045%2F101048909-505a1180-3583-11eb-945e-635f442c4961.png&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952080565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jcLuP8mGf8REMdOj83GCU4ekegXIKFArjHfbmVHySZo%3D&reserved=0 The distributions of confirmed and not confirmed candidates are in a very similar range: Comfirmed candidates(041): Median=5598.78, Mean=7441.88(+-9090.58) Not confirmed candidates(871): Median=5605.33, Mean=9153.18(+-10031.29) [image: mhcnuggets_boxplot] https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F16796045%2F101049211-9b742480-3583-11eb-90e5-48afc721ad15.png&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952090555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NZdRBpnFXVNuk%2Fh9IS1%2BFVnXVRO4p5dmzZEG3oQ2p8w%3D&reserved=0 When mhcnuggets scores are normalized (with added minus sign since smaller ic50 is better) and compared against VALIDATED outcome here is the output comparison figure: [image: mhcnuggets_plot] https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F16796045%2F101049847-2d7c2d00-3584-11eb-84d3-115c165d9dad.png&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952090555%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rkYGyO2kvCUU%2FxOnU4QoS%2F0NBRYAoqw9WjYp4Hj9y%2F0%3D&reserved=0
Here is the table with obtained scores https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.google.com%2Fspreadsheets%2Fd%2F1OO4k3xDjJtR_SRB603OF9XezXyOgmivcH255i8ecIL8%2Fedit%3Fusp%3Dsharing&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952100550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=N%2FYVhnc2kNEz3NLSwu1du%2FHkPlr3wN4FOJVrXLeB4pc%3D&reserved=0 as a reference and command used for one of the mhcnuggets runs: python /mhcnuggets/mhcnuggets/src/predict.py -c I --allele HLA-C05:01 --peptides merged.fasta -o mhcnuggets_HLA-C05:01.peps
Are these results expected? Could it be that this use case is out of the scope for mhcnuggets or it requires some additional tuning?
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FKarchinLab%2Fmhcnuggets%2Fissues%2F23&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952100550%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oS7gOrWEbPWW6NfKEJEFdGUl54brQVX9r0we9yGTpQE%3D&reserved=0, or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADC2PVB4WTDYZ6YT3N7V6BTSS6XBFANCNFSM4UMFAH5A&data=04%7C01%7Crkarchi1%40jhmi.edu%7C3916bb704e5141937a3208d897a272c9%7C9fa4f438b1e6473b803f86f8aedf0dec%7C0%7C1%7C637426073952110540%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iYcNU%2Fw1ySwfUghcIRsLHkndPzn2BnOZz3%2BW0Y%2F6x8g%3D&reserved=0 .
-- Rachel Karchin, Ph.D. Professor of Biomedical Engineering, Oncology and Computer Science Institute for Computational Medicine Johns Hopkins University 217A Hackerman Hall. 3400 N. Charles St. Baltimore, MD 21218
vladimirkovacevic
I was very curious to test the performance of mhcnuggets 2.3 on the latest dataset with TESLA validated neoantigen candidates published in Cell (Key Parameters of Tumor Epitope Immunogenicity Revealed Through a Consortium Approach Improve Neoantigen Prediction, Table S4 and S7). In TESLA they used flow cytometry and microscopy to confirm (VALIDATE) neoantigen candidates. More info is available in the paper. Here are ic50 scores obtained by mhcnuggets.
The distributions of confirmed and not confirmed candidates are in a very similar range:
Comfirmed candidates(041): Median=5598.78, Mean=7441.88(+-9090.58)
Not confirmed candidates(871): Median=5605.33, Mean=9153.18(+-10031.29)
When mhcnuggets scores are normalized (with added minus sign since smaller ic50 is better) and compared against VALIDATED AUC score is 0.513, precision is 0.0596, here is the output comparison figure:
Here is the table with obtained scores as a reference and command used for one of the mhcnuggets runs:
python /mhcnuggets/mhcnuggets/src/predict.py -c I --allele HLA-C*05:01 --peptides merged.fasta -o mhcnuggets_HLA-C*05:01.pepsAre these results expected? Could it be that this use case is out of the scope for mhcnuggets or it requires some additional tuning?