Questions about high power classifications

drphilmarshall / SpaceWarps

Science Team Website Development and Analysis

MIT License

12 stars 18 forks source link

Questions about high power classifications #201

Closed drphilmarshall closed 9 years ago

drphilmarshall commented 9 years ago

In the test in Paper 2 section 7 where we only use classifications with \Delta log P > 1.2:

Is there a typo, or do we really ignore all negative kicks?
What's the False Positive Rate with this cut? You can reach arbitrarily high completeness if you are willing to let the purity decrease to very low levels.

Also:

What fraction of classifications are retained with this cut?
How many subjects don't get any classifications at all with this cut?
How many unique agents contribute to the final sample?

drphilmarshall commented 9 years ago

Some of my confusion about purity comes from this sentence, I think:

"After testing on a subsample of both Stage I and Stage I FPs, we find that a small fraction of FPs are rejected when [only high power classifications are used]."

I think it's important to quantify our strategy in terms of purity and completeness (or TPR and FPR, but astronomers are more familiar with the former). Can we estimate these numbers please?

drphilmarshall commented 9 years ago

In #203 I re-word the recommendation somewhat - I don't think we necessarily need to give more numbers in this paper, but we'd want to make some purity/completeness or ROC curve plots before deciding what to do in the next project. This could be a good section for the eSWAP paper: @anupreeta27 if you want to issue that to Chris, based on the short experiment you describe in paper 2, please do just go ahead!

anupreeta27 commented 9 years ago

@drphilmarshall yes, agreed that in paper 2, we have preliminary results and how we exactly implement dynamic assignment in the next project needs further investigation. i'll issue chris on this.