QA execution and evaluation - Githubissues

microbiomedata / metaPro

Workflow for meta-proteomics analysis

BSD 2-Clause "Simplified" License

6 stars 3 forks source link

QA execution and evaluation #27

Closed pdpiehowski closed 1 year ago

pdpiehowski commented 2 years ago

This issue is to verify that the NMDC metaP pipeline produces the expected results. In this sprint, we will run the evaluation of the MetaP pipeline vs the previously established criteria and benchmark dataset to ensure quality of the output. The deliverable will be a summary of the metaP pipeline performance vs the established EMSL benchmark dataset. This evaluation is needed to finalize the metaP pipeline version 1. This work will be finished in sprint 2.

QA criteria-

95% peptide ID level overlap between DMS (EMSL current standard) and NMDC MetaP pipeline output
98% protein ID level overlap between DMS (EMSL current standard) and NMDC MetaP pipeline output
Correlation of log(2) abundances between the two approaches of 0.97 (Pearson correlations) Pipeline reproducibility
run the pipeline 5 times to see ensure reproducible output
- 95% peptide
- 99% protein
- 0.99 Pearson Correlation

pdpiehowski commented 2 years ago

Our first pass at QA evaluation did not match the QA criteria standards. We identified a number of small differences in the parameters and workflow. We have completed all the changes, and are re-running the processing now.

ssarrafan commented 2 years ago

Moving this to the next sprint to allow for QA process to complete. @pdpiehowski

ssarrafan commented 2 years ago

@pdpiehowski was this completed? Can this be closed? Did it pass QA?

@aclum FYI

ssarrafan commented 2 years ago

@pdpiehowski I'm moving this to the new sprint but please let me know if it's already done (I think it may be?) or if you won't be working on it the rest of this month. @aclum

pdpiehowski commented 2 years ago

Thanks Set, we moved even closer to passing the QA, but still seem to be missing an important detail. Progress will be a little slower while Sam is out, but we should be able to straighten it out in the next sprint.

ssarrafan commented 2 years ago

@pdpiehowski is this issue ready to be closed yet?

pdpiehowski commented 2 years ago

@ssarrafan still working, but found a bug that we think is responsible for the differences in outputs. We will know if that is the fix before the Thanksgiving break.

ssarrafan commented 2 years ago

@pdpiehowski moving this over to the next sprint hopefully one more time.

pdpiehowski commented 1 year ago

@ssarrafan Will need to move to the next sprint again.

ssarrafan commented 1 year ago

@pdpiehowski can this be closed?

ssarrafan commented 1 year ago

@pdpiehowski can we close this?

pdpiehowski commented 1 year ago

@picowatt is working on the code conversion for the FDR calculation and will be updating that soon. We will need one more sprint to knock this out finally.

ssarrafan commented 1 year ago

@pdpiehowski reported today that version 1 of the pipeline is done and they will have results of QA analysis in the next day or two so moving this to current sprint so it can be closed when QA is complete