Paper 1: how was purity estimated?

[drphilmarshall]

From @anupreeta27 by email:

I think the text that describes how the purity is estimated should also include the rough numbers for the expected number of lenses and non-lenses that went into the calculation, and how well-motivated they are (with appropriate references, if available). It would be great if we can add this to the revised version of the paper.

Agreed! :-)

[drphilmarshall]

The text of sw-system.tex currently reads as follows (line 1243 onwards):

The purity depends on the proportion of sims to duds, and so the purity of the test set must be approximated by rescaling the training set to the expected proportion of lens systems to not-lens systems in the survey. First we compute the expected number of false positives by multiplying the FPR by the expected number of non-lenses in the survey. Then we multiply the TPR by the expected number of lenses in the survey, to get the expected number of true positives. The sum of the true positives and the false positives gives the expected sample size; dividing the expected number of true positives by this sample size gives the purity. Note that the completeness is invariant to this transformation. The \StageOne curves are truncated by the retirement of subjects in this phase, which sets the minimum size of this sample. We see from the solid blue curve that over 90\% completeness was reached, albeit in a sample with 30\% purity.

@cpadavis, can you please report on the following numbers that you used?

• "expected number of non-lenses in the survey"
• "expected number of lenses in the survey"
• "the expected sample size"

Thanks!

[cpadavis]

So about purity. We take as given that TPR and FPR are the same between our training system and the survey data. But purity = TP / (TP + FP), not the rates. In order to get the purity estimation of the CFHTLS lens sample from our training, where the proportion of lenses and duds is not 50-50 but instead 2-9998, we need to rescale the training FP and TP to what we expect to see in the wild. So in other words, FP_wild = FPR * (1 - lens fraction) * sample_size, TP_wild = TPR * lens_fraction * sample_size, so our expectation for purity in the wild is TPR * lens_frac / (TPR * lens_frac + FPR * (1 - lens_frac)) (sample size divides out). So in effect the only number I used is the lens fraction, which was just our prior on lens probability, 2e-4.

I remember coming up with this explanation with Phil to help make it clear how we obtained the purity from the SpaceWarps system. We can rewrite it if we think this'll make it a little more clear what we did.

[anupreeta27]

@cpadavis @drphilmarshall i wanted to ask this in the telcon but forgot again - the abstract and pg. 12 - ref. fig. 7, we say over 90% completeness for stage 1 which I can see in the plot but I see a purity of 15%. for the "detection" limit. how do we get 30% purity ?

[cpadavis]

The inset figure shows that 92% completeness at 30% purity. The star is for the 95% probability, at which point we have 94% completeness but only 15% purity. (The only difference between the two is where we set the detection threshold. You'll notice a similar sort of story in the ROC curve, where a lower detection threshold actually can improve the true positive rate for minimal loss in false positive rate.)

On Wed, Jul 22, 2015 at 7:03 PM anupreeta27 notifications@github.com wrote:

@cpadavis https://github.com/cpadavis @drphilmarshall https://github.com/drphilmarshall i wanted to ask this in the telcon but forgot again - the abstract and pg. 12 - ref. fig. 7, we say over 90% completeness for stage 1 which I can see in the plot but I see a purity of 15%. for the "detection" limit. how do we get 30% purity ?

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/218#issuecomment-123929889 .

[anupreeta27]

right but our sample was for P>0.95, so that would mean the purity is 15% or lower (it can't be better than that ) at stage 1

[anupreeta27]

just to explain further - the abstract and the text on page 12 implies that stage 1 "has" 30% purity. you could either say "can have / reach" 30% purity or you can say "it has 15% purity"

[drphilmarshall]

Good catch - I'll fix this while I'm adding Chris' numbers.

On Thu, Jul 23, 2015 at 8:34 AM, anupreeta27 notifications@github.com wrote:

right but our sample was for P>0.95, so that would mean the purity is 15% or lower (it can't be better than that ) at stage 1

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/218#issuecomment-124141953 .

[cpadavis]

Agreed -- good catch @anupreeta27!

[drphilmarshall]

OK, I fixed the purity discussion in https://github.com/drphilmarshall/SpaceWarps/commit/0eb38cce5e0fea7034166a0216cd4ba859077ce3