Stage 2 Re-classification of eSWAP CFHTLS Stage 1 candidates

[drphilmarshall]

Hey @cpadavis,

As I said just now over tea, opportunity knocks! Can you prepare a list of CFHTLS stage 1 subject Zoo IDs for everything you'd like stage2-classified please? We're resurrecting the website this summer to give the crowd something to do after reading about our CFHTLS papers, and it's a good opportunity to try and find a few more lenses based on your eSWAP analysis. See #214 for the whole plan!

• [ ] In the eSWAP paper latex file, make very brief, itemized notes on the questions you want to answer, defining terminology for the various samples you want to investigate following the stage 1 offline SWAP runs you have already done.
• [x] Make the samples, writing out the Zoo IDs (and any other useful information you want to keep together) to some clearly named plain text catalog files.
• [x] Ship these samples to Anu, along with instructions as to which sims and duds you want re-activated as well (I think you said that all of them was better than some of them, and you wanted the stage 1 training set not the stage 2 one - to include the lensed quasars).
• [x] It sounds like the CFHTLS Stage 2 classification interface is the one we should use to do this refinement exercise, but we can discuss this here.

Thanks for taking care of this before you go off on tour for three weeks with the Stanford Jazz Orchestra! We'll keep in touch via this issue thread. Have fun!

Phil

[drphilmarshall]

Hey @cpadavis ! Hope the tour is going well :-) Any progress on the sSWAP candidate list?

[drphilmarshall]

We realized we need equal numbers of sims and duds (since the web app just draws a training subject, without looking to see if its a sim or a dud), so we'll just use 450 sims to match the 450 duds we have. We'll make sure that we use equal numbers of lensed quasars, lensed galaxies and cluster lenses.

[cpadavis]

Bonjour from Paris! Sorry about the delay: the internet at the hotel is terrible which makes working on here a bit of a hassle. (Mostly with regard to plots.) Let me give you guys an update on what's up.

I have a catalog made at http://www.slac.stanford.edu/~cpd/targets/targets.tar.gz

In that directory are several csv's as well as some plots. These were generated from make_eswap.py, which currently is living in my branch of the spacewarps repo. I'm hoping with our dataset to get a closer look at the targets which disagree greatly between the online and offline systems. I'm also curious about the targets that disagree greatly between online supervised and offline supervised plus unsupervised (so using the test targets alongside the training to better determine the PL and PD). It turns out that the vast majority of 'great disagreements' are ones where the offline system believes that an object should be a lens while the online system thinks it is a dud.

Each csv file has the following columns: ID,ZooID,mean_probability,kind,knownlens,logit_prob,logit_prob_cut,logit_prob_online,logit_prob_cut_online,mean_probability_online,logit_prob_alldata,logit_prob_cut_alldata,mean_probability_alldata

ID and ZooID are self-explanatory. kind is sim, dud, test, while knownlens is a boolean for whether it is a known lens (before SpaceWarps). mean_probability is the probability from the supervised offline system (ie the one we used in stage 2), mean_probability_online is from the online system, and _alldata is the offline supervised plus unsupervised. logit_prob is the logit transformation of the probability data, while logit_prob_cut is the logit probability cut off at +-5 (0.993 and 6.69e-3 respectively).

These are the csv files:

knownlens_stage.csv - known lenses in the spacewarps sample. We want all of these because they are really our main way of assessing how the program performs on real lenses. dud, sim, train - all of the duds, sims, and duds and sims in the stage. high_chance_stage.csv - offline > 0.35, online < 0.01. I choose a relatively low probability cut here because we found that in the roc curves the 'best' cutoff (for maximizing TPR and minimizing FPR) was around 0.4 in both systems, so really things with p > 0.35 are potentially very interesting. There are 1594 objects here. high_chance_unsup_stage.csv - offline unsupervised + supervised probability > 0.7, online < 0.01. The higher cutoff here is because the unsupervised probabilities tend to be higher when the online system is unsure (see plots below). There are 1808 objects in this catalog. Between this and the other high_chance, there are 2057 unique objects.

That gets us something like 2200 objects. I can tune the probabilities to get more or less objects as we feel necessary. What do you guys think?

[drphilmarshall]

Thanks Chris! The Stage 1 detection threshold was P_online = 0.95: only subjects with this Stage 1 probability or higher made it into Stage 2, so I'd change your "online < 0.01" to "online < 0.95" I think. Then, the difference in the offline thresholds between your high_chance_unsup and high_chance samples looks a bit odd, but if you are extracting these values in some principled way from the training set ROC curves, that's fine.

Good news that the disagreements are almost all potential new detections! I think between your two samples and Anu's "unlucky" subjects, we should have enough for a meaningful test.

Re sims and duds, we realized that we have to use only 450 sims, because we only have 450 duds (and the system needs equal numbers of each). I hope this will be enough for a good eSWAP Stage 2 ROC curve - what do you think?

[cpadavis]

Thanks Phil for the comments. 450 sims and duds should be fine. Updating the threshold from 0.01 to < 0.95 adds about one thousand more candidates, which is fine. I will update the catalog tar probably before I go to bed tonight and will make a post when I do.

[cpadavis]

OK I have made a csv that just contains the known lenses and the 'high chance' objects described above (for online < 0.95). It can be downloaded at: http://slac.stanford.edu/~cpd/targets/targets_stage1.csv

[drphilmarshall]

Excellent, thanks Chris! @anupreeta27, can you download this catalog and check that it contains everything you need, please?

On Sun, Jul 12, 2015 at 2:07 PM, Chris notifications@github.com wrote:

OK I have made a csv that just contains the known lenses and the 'high chance' objects described above (for online < 0.95). It can be downloaded at: http://slac.stanford.edu/~cpd/targets/targets_stage1.csv

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/215#issuecomment-120762375 .

[drphilmarshall]

@anupreeta27 tells me she should be able to check out your catalogs and if all is OK, pass them to @chrissnyder in the next few days. Heads up, Mr. Snyder! We should have some data to upload to the SW website very soon, so that we can be live when the papers are accepted. (See #214)

[anupreeta27]

thanks @cpadavis and @drphilmarshall, i will pass on the "targets_stage1.csv" catalog. i notice that the types of some of the subjects is "sim" - not sure why they are there. are sims meant to be in there? could it be that these have the "known lenses" but also have sims ? In that case, I will just pick the corresponding image without the sim in it. Please let me know.

@chrissnyder if i send you a catalog with the following columns, would you be able to reactivate the corresponding subjects? e.g. 5183f151e4bb2102190061fd CFHTLS_094_1957 ASW0000jct

thanks

[cpadavis]

Hi @anupreeta27: yes, there are objects that have sims that also have known lenses, for example

https://static.zooniverse.org/spacewarps.org/subjects/standard/5183f151e4bb210219037f4b.png CFHTLS_074_2145 ASW0004wuj

(there is the obvious sim bottom middle, but a lens middle top. The known lens is not in the alpha layer)

I just selected all objects that contained known lenses, so if we could put in the corresponding image without the sim, that would be fantastic.

[drphilmarshall]

I think this issue can almost be closed, if @anupreeta27 is happy with the dataset. Looking at the to-do list above: I checked off the choice of interface, since we decided offline to go with the blue Stage 1 interface this time (we're just accumulating more classifications using the standard Stage 1 training set). Then, the last thing that needs doing is updating the eSWAP latex with the tests that are being performed here ("Three samples of candidates were selected for further inspection ... These were re-ingested into the SW web app and ..." etc etc.) Over to you, @cpadavis ! :-)

[anupreeta27]

i am fine with closing this. i will email @chrissnyder to check if he needs more columns.

[anupreeta27]

@cpadavis hi chris, could you please forward a catalog of subject IDs which are labelled as duds from the stage 1 (online) pickles (actually, there's just one duds catalog used ever consisting of 450 subjects) ? we will need these along for the SW reboot as well. thanks!

[ccld]

There are also some known lenses not tagged as such in high_chance: ASW0009clg ASW0005udz ASW0006hoa ASW00056pq ASW0002lcz

Please check and update

[cpadavis]

@anupreeta27 do you just want a random selection of 450 duds? (I have 4500 in stage 1.) I don't think I have any preferences on which duds are used. One with the 450 highest probabilities (= the hardest) are now at http://slac.stanford.edu/~cpd/targets/dud_worst450_1.csv

@ccld: (I'm assuming you mean http://slac.stanford.edu/~cpd/targets/targets_stage1.csv and not the older high_chance_1.csv that was in a tar.) I went through those, and the latter three are marked as known lenses:

                              ID  mean_probability  kind knownlens  \

ZooID ASW0009clg 5183f151e4bb21021906a804 0.419572 test False ASW0005udz 5183f151e4bb210219042917 0.371980 test False ASW0006hoa 5183f151e4bb210219049efa 0.393704 test True ASW00056pq 5183f151e4bb21021903b13e 0.392782 test True ASW0002lcz 5183f151e4bb21021901d8a3 0.373378 test True

        mean_probability_online

ZooID ASW0009clg 5.849845e-03 ASW0005udz 9.968599e-08 ASW0006hoa 3.978971e-05 ASW00056pq 7.637959e-04 ASW0002lcz 1.156056e-04

The first two are not in my known lens catalog that I was comparing against -- so if they are known lenses, then my known lens catalog is incomplete! Could I have your list of known lenses?

[cpadavis]

Oh and @anupreeta27 the complete list of stage 1 duds can be found at http://slac.stanford.edu/~cpd/targets/dud_1.csv

[drphilmarshall]

That's great, thanks @cpadavis! I suspect that your complete list contains the same 450 subjects over and over again - but that's good, because we need equal numbers of sims and duds for the web app to select randomly from. @anupreeta27 are we all set now?

On Fri, Jul 31, 2015 at 6:25 AM, Chris notifications@github.com wrote:

Oh and @anupreeta27 https://github.com/anupreeta27 the complete list of stage 1 duds can be found at http://slac.stanford.edu/~cpd/targets/dud_1.csv

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/215#issuecomment-126690664 .

[ccld]

@cpadavis It was high_chance_1.csv. The first two ASW0009clg and ASW0005udz are to be checked with @anupreeta27. I commented months ago on talk known reference SA115 and SA98.

[anupreeta27]

@cpadavis @drphilmarshall is right. the sample of 4500 is just a repeat of the 450 duds. Is it possible you to select the first 450 unique duds (since the launch) from the pickles because that will give us the easiest way to select the unique duds sample. I doubt there's any other way of distinguishing which 450 correspond to which dataset (just fyi, we had split the entire cfhtls data in to 10 diff. datasets with dud images being repeated in all of the datasets but with a different ID)

[cpadavis]

Hi @anupreeta27 and @drphilmarshall : naive question: without actually looking at them, how can I tell which duds are the unique ones? They all have different ID and ZooID parameters (and so also point to different image files). I could imagine though that either the first 450 or every tenth image (ordered by zooid) is unique. When I have better internet I will take a look and get you that catalog.

[cpadavis]

OK it looks like they were ordered by sets of 450 when sorted by ZooID. I took the first 450 (sorted by ZooID) and put them here: http://slac.stanford.edu/~cpd/targets/dud_probably_unique_1.csv

I haven't actually looked at all 450 images (hence 'probably' in the title), but I did check that each ZooID here corresponded to a unique CFHTLS field

[anupreeta27]

@cpadavis whatever you did is correct. that's exactly what i wanted :) thanks !