search

drphilmarshall / SpaceWarps

Science Team Website Development and Analysis
MIT License
12 stars 18 forks source link

Resurrect SW CFHTLS Stage 1 website so we are live when publicity goes out #214

Closed drphilmarshall closed 9 years ago

drphilmarshall commented 9 years ago

Things that will need doing before ~Friday 24th July:

aprajita commented 9 years ago

perhaps we should colour the interface to some other colour (SW green?) to show it's a different process to stage 1 &2

drphilmarshall commented 9 years ago

Change of plan: no new sims in the next 3 weeks but we can do tests in CFHTLS to try and recover some false negatives instead. Editing the check list at the tope of this thread!

ccld commented 9 years ago

I wonder what are your " "unlucky" CFHTLS Stage 1 subjects?

Having a quick look at "high_chance_1.csv" from today Chris csv files, I find for example same object: ASW0009d7f and ASW0009d7g tagged #lens #arc which are not in stage 2 but I don't know why by your probabilities.

Have you any explanation?

cpadavis commented 9 years ago

I should have been more clear about the meanings of the columns: mean_probability was from the 'offline' system, which attempts to determine the user skill and lens probability simultaneously. Stage 1 uses the 'online' system, which updates lens probability with each classification and user skill with each training classification. The column that corresponds to what was used in stage 1 is mean_probability_online, which gave them very low probabilities. This is because at the time of evaluation most of the users who evaluated those two objects had middling PLs (probability to find a lens given lens) and high PDs (probability to find a dud given that it is a dud). Further, more people evaluated them as duds. Below I print out the PDs of those who classify the object as a dud and the PLs of those who classified it as a lens (these are those values at the time of classification, so what went into the online system):

ASW0009d7f PD [ 0.9898149 0.98990246 0.99 0.99 0.99 0.99 0.9898027 0.99 0.94860434] PL [ 0.81772108 0.61290323 0.28571429 0.5 0.4 0.87102178] ASW0009d7g PD [ 0.9898149 0.76470588 0.98990246 0.99 0.99 0.98990706 0.98991022 0.97853032 0.97674419 0.93076923 0.6 0.98972836 0.94861573 0.98860887] PL [ 0.75903614 0.33333333 0.79686839 0.53333333 0.7745098 0.91959448 0.83996212]

You can see that the system thinks most people who classify objects as duds are very good at it, but isn't so confident about the ability for people to find lenses. Couple that with the fact that more people classify the objects as duds than lenses, and you obtain the low lens probabilities indicated in the mean_probability_online column.

In the offline system, we try to simultaneously determine the user skill (PL and PD) with the probabilities. The PLs and PDs we obtain at the end of the online process don't actually differ that much from the ones obtained in the offline. The main difference is that we use those final values to make the classification. It looks here like the main difference is that the PL increased, so users who vote 'lens' counted for more in the offline than the online in those two cases:

ASW0009d7f PD online final - PD at evaluation [ 0.00000000e+00 -1.11022302e-16 0.00000000e+00 0.00000000e+00 -2.94464075e-04 0.00000000e+00 -1.11022302e-16 0.00000000e+00 1.13833129e-05] PD offline final - PD at evaluation [ 0.0022486 0.00344418 0.00537547 0.00575822 0.00823113 0.00663474 0.00731417 0.00639581 0.00021019] PL online final - PL at evaluation [ 0. 0.017866 0.24369748 0.22619048 0.08387097 0.01050689] PL offline final - PL at evaluation [ 1.11091286e-04 2.20174091e-02 2.47619048e-01 2.31707317e-01 8.27586207e-02 1.14814173e-02] ASW0009d7g PD online final - PD at evaluation [ 0.00000000e+00 0.00000000e+00 -1.11022302e-16 0.00000000e+00 0.00000000e+00 9.29368030e-05 8.97808292e-05 1.11022302e-16 1.21396752e-02 2.09992580e-02 6.66666667e-02 2.71637816e-04 0.00000000e+00 1.39112903e-03] PD offline final - PD at evaluation [ 0.0022486 0.03529412 0.00344418 0.00537547 0.00778221 0.00784827 0.00375746 0.00021193 0.01720438 0.02392333 0.15 0.00690638 0.00019881 0.00745412] PL online final - PL at evaluation [ 0.02995468 0.30952381 0.0259346 0.21666667 -0.01368118 0. -0.00322743] PL offline final - PL at evaluation [ 3.53563788e-02 3.33333333e-01 2.61201181e-02 2.19191919e-01 -1.26950591e-02 1.46737009e-04 -2.67675735e-03]

Hope this helps!

drphilmarshall commented 9 years ago

Alright, some significant progress on resurrecting th eweb app - I'm tracking the merge/combine of all our branched off code here. @aprajita, I think the Spotter's Guide reads OK, but great if you could fire up a local instance and give it a read through to check that I have all the text correct. It's all now in the same translations/en_US.txt file, where I've put in some CFHTLS and VICS82 flags where necessary.

Once Chris Snyder has helped us fix the landing page css we should be good to merge back into our Lens-Zoo master branch (at last!) and then submit a pull request to the Zooniverse.

drphilmarshall commented 9 years ago

Last thing for today (the rest of https://github.com/drphilmarshall/Lens-Zoo/pull/77 needs @chrissnyder's input): I added a link to this repo on the About page, in this commit - hope that looks OK to you!

aprajita commented 9 years ago

@drphilmarshall have local version that all looks fine from perspective of SG. Also get the same error on the landing page you just mailed Chris S about.

I am not sure we had to remove everyone not on the papers from the 'Team' pages... in fact I would think that everyone who contributed at any level is listed here, whether they contributed sufficiently for the papers is another matter.

aprajita commented 9 years ago

@drphilmarshall We also need a message on the landing page or popping up at the first classification that says this is a rerun - as the explanation at the bottom of the about page is pretty hidden

drphilmarshall commented 9 years ago

OK, cool - the people page is, I guess, the only place where we introduce people who are on Talk, so listing everyone as "science team" makes sense. I'll fix this.

I'll add a note about the nature of the task on the landing page too - having the content there should help Chris out when he fixes the css. Thanks!

On Fri, Jul 31, 2015 at 7:09 AM, aprajita notifications@github.com wrote:

@drphilmarshall https://github.com/drphilmarshall We also need a message on the landing page or popping up at the first classification that says this is a rerun - as the explanation at the bottom of the about page is pretty hidden

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/214#issuecomment-126704227 .

aprajita commented 9 years ago

thank you for doing all of this!

drphilmarshall commented 9 years ago

OK, check it out: the about page now lists everyone on the CFHTLS project, with zooniverse usernames where they exist. And i edited the "explanation" field on the landing page to show we are doing a re-run. Can you please take a look and edit it if you can think of better words? Thanks!

aprajita commented 9 years ago

reads fine - thanks i think for people skipping the landing page is there a way to add a pop-up at the beginning of a classification session? e.g. will everyone have to go through the tutorial again? if so we could just add an introductory message in bold to the top of the first page of the tutorial window - this way they will at least see that before they hit 'x'

On 31 Jul 2015, at 15:45, Phil Marshall notifications@github.com wrote:

OK, check it out: the about page now lists everyone on the CFHTLS project, with zooniverse usernames where they exist. And i edited the "explanation" field on the landing page to show we are doing a re-run. Can you please take a look and edit it if you can think of better words? Thanks!

— Reply to this email directly or view it on GitHub.

drphilmarshall commented 9 years ago

I think only new users will see the tutorial - so I edited the first message to reflect the reboot exercise. This was a very good catch actually - I hadn't captured all our old feedback and tutorial options yet! Thanks A.

aprajita commented 9 years ago

thanks Phil - pls let me know if i can do anything

On 31 Jul 2015, at 16:42, Phil Marshall notifications@github.com wrote:

I think only new users will see the tutorial - so I edited the first message to reflect the reboot exercise. This was a very good catch actually - I hadn't captured all our old feedback and tutorial options yet! Thanks A.

— Reply to this email directly or view it on GitHub.

drphilmarshall commented 9 years ago

I don't think we can do much more until @chrissnyder springs into action! But then we'll need to test the tutorial and feedback messages in action.

On Fri, Jul 31, 2015 at 9:28 AM, aprajita notifications@github.com wrote:

thanks Phil - pls let me know if i can do anything

On 31 Jul 2015, at 16:42, Phil Marshall notifications@github.com wrote:

I think only new users will see the tutorial - so I edited the first message to reflect the reboot exercise. This was a very good catch actually

  • I hadn't captured all our old feedback and tutorial options yet! Thanks A.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/drphilmarshall/SpaceWarps/issues/214#issuecomment-126743256 .

drphilmarshall commented 9 years ago

Closed via https://github.com/drphilmarshall/Lens-Zoo/pull/77!