How do I check conventions for and enable feedback on Space Warps Demo?

[aprajita]

https://github.com/zooniverse/Panoptes-Front-End/pull/3202 has been merged so am starting a new issue as @srallen advised. This issue is part of the SW feedback requested in https://github.com/zooniverse/Panoptes-Front-End/issues/2834 (now closed).

Expected behavior

Same numbering is used in current behaviour 1) Expect to see a success message for SIMs, but failure message shown even when marker appears to be in the right place 2) Not sure how to implement feedback in the SWDemo site

Current behavior

Please include any error messages from the browser console and/or screenshots

1) I uploaded a selection of SW subjects including sims & duds onto https://space-warps-feedback.pfe-preview.zooniverse.org/projects/hughdickinson/testing-metadata-based-feedback/classify to see how this works.

The success and fail messages seem to be OK for duds, but for sims the success message doesn't appear always giving a failure message (see fig. below). I suspect this is something in to do with either the starting point from where the x,y positions should be defined and/or the tolerance. We include the "radius" for tolerance around the lensed feature's x,y position in the manifest under "Tol". As I can't see the manifest for @hughdickinson 's test dataset, I can't see what the required format/keywords are to provide this info. Please could you let me know what the conventions for this info should be?

image this is for image_4.png showing the failure message and marker on the feature. I expected the feature to be marked around (165, 269) and gave a generous tolerance of 80 so a success message should have been recorded.

2) Also if this functionality is now merged, I'm not quite sure how to select this functionality under our SWDemo workflows (https://www.zooniverse.org/lab/1434) - please could you advise?

Steps to replicate

Additional information

• Operating system:
• Browser:

[srallen]

@hughdickinson could you share the manifest for the test data for the metadata feedback so we know how the subject metadata needs to be formatted for this feature? Thanks.

[aprajita]

@hughdickinson i was wondering if you might be able to share the manifest with me as the feedback didn't seem to work well ( query 1 above)?

Also can you advise me on query 2 above?

Many thanks!

[aprajita]

@hughdickinson @srallen would someone please be able to advise me on either of these queries when you have chance. we would like to launch the new space warps project with feedback working soon, once the data is ready. i believe @marten also will need to do some NERO testing before we are ready to go. many thanks!

[lcjohnso]

Hi @aprajita and others -- I'm Cliff, and I have experience with feedback features through my work with the Andromeda Project (AP). I'm interested in helping here because I too would like to enable and test metadata-based feedback in a new star cluster finding project I'm hoping to launch in the next couple months.

Regarding Query 1: I believe that the X/Y coordinate systems a) used by the markers, and b) used in the metadata for the true SIM location (#X, #Y) are using different conventions for the origin -- markers are references from top-left corner, but most astronomers are used to a bottom-left origin. This was a gotcha we encountered with AP and seems consistent with your non-detection in "image_4" above. --Possible solution: upload a new manifest and/or new subject set to fix the #X/#Y coordinate convention such that it matches the one used by the markers. --Possible solution: to determine what manifest content was provided for a subject set, you could download the *subjects.csv file from the project's Data Exports page and look at the key values used in the "metadata" JSON entry. You can filter the file by subject_set_ids to isolate a particular subject set in case that file is large.

Regarding Query 2: My understanding is that like all experimental features, an admin needs to enable this option on a project-by-project basis. I'm not sure who can do this (@srallen?), but I'd also like to help test the feature using a cluster test project I built and a test workflow that is ready to go. --Action Item: Could a zooniverse admin flip the switch for feedback-based-metadata for AP AAS229 Test Project (ID = 3627)?

I hope this helps!

[srallen]

@lcjohnso We're experimenting with using subject metadata for the feedback mechanism and what we have built for Space Warps isn't generalized at the moment, so yes, it is an experimental feature we could enable, but I don't believe it may exactly work for what you need. @rogerhutchings is going to be working on making it a generalized tool.

@aprajita Regarding the metadata example, I'll get in contact with Hugh to see if we can get more information, but in the meantime, I can look into getting the subject export from the test project which will have the metadata used that was used for it. Thanks for the suggestion @lcjohnso

[srallen]

@aprajita here is the subject export for the test project. The metadata field is in JSON, so let me know if you need any assistance with parsing that.

https://drive.google.com/file/d/0B9PLKSIl-AdPR2h2akY0eG5iVlU/view?usp=sharing

[srallen]

@aprajita Here's the manifest from the test project. Let me know if you have any more questions.

https://drive.google.com/file/d/0B9PLKSIl-AdPN1dHdjFKZHJpb1E/view

[aprajita]

Hi @lcjohnso & @srallen Many thanks for your replies and for the manifest.

Indeed the coordinates are defined from bottom left, and suspected this might be the source of the problem. So 0,0 top left, the X increases downwards and y to the right, must admit this feels counter intuitive but I guess there are reasons to define them that way.

What's the zooniverse recommendation, to modify our frame to match the markers, or to modify the code for the feedback?

Thanks very much! Aprajita.

[lcjohnso]

So 0,0 top left, the X increases downwards and y to the right

@aprajita Actually, Y increases downwards and X increases to the right, (0,0) in top left. See this image as an example.

[srallen]

@aprajita we use SVG in the classifier and SVG coordinate space starts 0,0 in the top left. We don't have a standard recommendation, it depends on what works for your data best. If the feedback message needs adjustment, let me know I can do that.

[aprajita]

sorry yes, x right, y down - duh - thanks @lcjohnso

@srallen we'll make sure we use your convention for the manifest (origin 0,0 top left) so will amend the manifest for the SW trial data and try again.

Thanks!

[srallen]

@aprajita While writing some tests, I discovered a line of code that doesn't work as intended with the test project: https://github.com/zooniverse/Panoptes-Front-End/blob/master/app/classifier/metadata-based-feedback.cjsx#L66

Do you need or want to give feedback to volunteers when they see a real subject? I could change that conditional to check for a 'SUB' type in the subject metadata for that case. Would you want to display a success message from the metadata or use what is hard coded, "You classified some real data!"?

[aprajita]

hi @srallen

thanks for spotting this.

We do not want feedback on normal subjects, only on those we use for training - these will be made up of simulations (SIM), empty fields (DUD), known lenses (KNOWN), false positives (FP). I'm just checking if we want to have all of those. The manifests will be populated with the success and miss messages for each type of training subject, rather than the system picking from a list of possible messages as we did in the old platform.

@marten can I check what NERO wants to know - does it just need a subject 'SUB' versus training 'TRAIN' flag or does it want to know if it's a sim/known vs. a dud/FP? In the former we can use 'TYPE' to hold just 'SUB' & 'TRAIN' and a subtype that holds the sub-categories for the training images.

I think for the front-end you still need to know if it's a SIM/KNOWN or DUD/FP sub-type as the trigger for the messages differs, i.e. success is a click within tolerance for SIM/KNOWN, and no click for DUD/FP.

[srallen]

@aprajita Ok, thank you for the information. I will need to modify the code to look for KNOWN as well as FP. Currently it is only setup to display messages for SIM and DUD. Could you send me a sample of subject metadata for each type so I can develop against it? Thanks.

[aprajita]

Hi @sralken before you start coding please could you wait until I reply with our final list of keywords and also if @marten can confirm regarding Nero I'll make the types and subtypes accordingly. As the actions for sim is the same as known and dud is the same as FP we can group these together so I to k leave it as it is now and I'll get back to you as soon as i can. Many thanks!

On Tuesday, 31 January 2017, srallen notifications@github.com wrote:

@aprajita https://github.com/aprajita Ok, thank you for the information. I will need to modify the code to look for KNOWN as well as FP. Currently it is only setup to display messages for SIM and DUD. Could you send me a sample of subject metadata for each type so I can develop against it? Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/3361#issuecomment-276399008, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1EEi2ZZrI1VHAePvBpKdrprTOm7QQTks5rX1ZDgaJpZM4LUd48 .

[srallen]

@aprajita I can definitely wait and I have a few higher priority tasks for this week, so when you're ready, please do share a sample set of metadata for me to work with. Thanks.

[aprajita]

Hi @srallen I've just amended the positions to be measured from a top left origin for the SW demo dataset I uploaded into @hughdickinson's test site, but I still don't get clicks on sims getting the success feedback message. I think it may have been the type metadata was set differently to huh but now have hit a max number of dataset uploads (100) so I can't add my amended subject set with manifest. Is there a way to increase this limit?

Also is there a way to update a manifest without having to upload the entire dataset again?

Many thanks Aprajita.

[srallen]

@aprajita could you share a small sample set of subjects and their metadata csv with me so I can take a look? I can increase your max upload limit too. What is your username on zooniverse.org?

Yes, there is a way to update subject metadata without reuploading. It would involve writing a script to directly interact with our API. We've built a python client to make this a bit easier and can be installed via pip. However, if the subject set is relatively small, it might just be easier to upload a new subject set and unlink the incorrect one from any workflows.

[aprajita]

Thanks yea please do increase my limit. My zooid is aprajita

The sample I am using is just a small subset for testing 16 subjects. I just uploaded, deleted and reuoaded too many times. I think this upload will work though so if you increase my quota from 100 to 200 that should do the trick.

Thanks Aprajita

On Wednesday, 1 February 2017, srallen notifications@github.com wrote:

@aprajita https://github.com/aprajita could you share a small sample set of subjects and their metadata csv with me so I can take a look? I can increase your max upload limit too. What is your username on zooniverse.org?

Yes, there is a way to update subject metadata without reuploading. It would involve writing a script to directly interact with our API. We've built a python client https://github.com/zooniverse/panoptes-python-client to make this a bit easier and can be installed via pip. However, if the subject set is relatively small, it might just be easier to upload a new subject set and unlink the incorrect one from any workflows.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/3361#issuecomment-276686648, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1EEtJFnZrFqLnu17sKzUEdSmAzAw12ks5rYKQdgaJpZM4LUd48 .

[srallen]

@aprajita I've increased your upload limit to 100,000 for production zooniverse.org and 1,000 for staging (the test project), so you shouldn't run into the limit for a long time.

[aprajita]

excellent thanks @srallen

On 1 February 2017 at 16:57, srallen notifications@github.com wrote:

@aprajita https://github.com/aprajita I've increased your upload limit to 100,000 for production zooniverse.org and 1,000 for staging (the test project), so you shouldn't run into the limit for a long time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/3361#issuecomment-276713848, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1EEq7b3DHU79nxo3u0zmd5TZlugjYGks5rYLlfgaJpZM4LUd48 .

[aprajita]

Hi @srallen I don't seem to have access to @hughdickinson's test site anymore. e.g. e.g. i can no longer view https://space-warps-feedback.pfe-preview.zooniverse.org/lab/1639/collaborators Please could you check? I logged out and in again but that doesn't seem to help. thanks

[eatyourgreens]

Hi @aprajita could you try that link again?

[aprajita]

yep - thanks @eatyourgreens works now!

@srallen trying out the new manifest didnt help though. It still gives me a fail message for SIMS even if the markers are placed and are within the tolerance. Could you please take a look? Shall I upload my dataset somewhere or can you access it from https://space-warps-feedback.pfe-preview.zooniverse.org/lab/1639/?

Many thanks both.

[aprajita]

the manifest & images are here https://drive.google.com/open?id=0Bwx2SzYNg-RmVHUtaFJVN1BLUmc

[srallen]

@aprajita Thank you for sharing the samples. I'll take a look when I can, however, there are a few higher priority issues I'll be looking at this week. I'll respond back here when I can.

[aprajita]

Hi @srallen. I've been looking at https://github.com/zooniverse/Panoptes-Front-End/blob/master/app/classifier/metadata-based-feedback.cjsx trying to figure out what's not working, but I can only think the meta data labels are inconsistent, I tried both replicating the manifest you sent me with #X1 & #Y1, as well as reverting back to #X & #Y, but both yield the same result that accurate user marks on the test subjects (sims) are not being registered as right but triggering a "missed" message.

Could you please take a look what's wrong so that we can implement this and start to test the interface with NERO.

The meta data labels I have will be a very slightly different to what's coded here, but i would like ot understand why this isn't currently working for the SW test dataset before implementing those changes.

Best wishes, Aprajita.

[aprajita]

hi @srallen again - i think the issue might be in
https://github.com/zooniverse/Panoptes-Front-End/blob/master/app/classifier/metadata-based-feedback.cjsx#L62

if (!(userMadeAnnotation?) || !isWithinTolerance) && subjectFailureMessage `

might work better for test subjects for SW if it was written as:

if (userMadeAnnotation && isWithinTolerance) && subjectSuccessMessage
        <p>{subjectSuccessMessage}</p>
      else if subjectFailureMessage
        <p>{subjectFailureMessage}</p>

(not sure if we need the && subjectSuccessMessage as all test subjects will have a success and failure message in the metadata)

thanks, a.

[aprajita]

Hi @srallen, could you please let me know when you might have time to look at this. I'd like to run a test with some volunteers with this feedback feature enabled and registered for @marten to check SW with NERO.

Many thanks! Aprajita.

[rogerhutchings]

Hi Aprajita, I'm happy to take a look at this, I'm working on feedback anyway - I should get to it before the end of the week.

[aprajita]

thanks @rogerhutchings, just realised that my web page wasn't updated when I wrote my last message and didn't see @srallen had referenced this issue in 3527 when i wrote. thanks to both of you for looking into this. if we can get this working with the test data, I'll then be able to make a test dataset with meta data labels that i hope will be close to the final subject set (slightly different to the set here which is just to test that the feedback from the original test works with SW). thanks again!

[srallen]

@aprajita Apologies for the delays. @rogerhutchings FYI this branch is the work I did to convert to ES6 and write tests, but didn't merge it because of the questions that came up here.

[aprajita]

hi @srallen do you happen to know where we are with this, i got an auto-message from @rogerhutchings to say he is on annual leave for a while but didn't hear if he had made any progress. thanks!

[trouille]

Hi @aprajita, @rogerhutchings will pick back up here when he's back from leave on March 24.

[aprajita]

@rogerhutchings please do get in touch when you can so that i can get an idea of how far off this is for planning purposes. many thanks.

[rogerhutchings]

Hi @aprajita, I'm taking a look at it now.

[rogerhutchings]

@aprajita can you take a look at https://metadata-based-feedback-fix.pfe-preview.zooniverse.org/projects/hughdickinson/testing-metadata-based-feedback/ and see if it's working as expected now?

[aprajita]

hi @roger - thanks from a quick go, i think yes, but will play around more later and send some comments

many thanks for looking into this!

On 31 March 2017 at 17:41, Roger Hutchings notifications@github.com wrote:

@aprajita https://github.com/aprajita can you take a look at https://metadata-based-feedback-fix.pfe-preview.zooniverse.org/projects/ hughdickinson/testing-metadata-based-feedback/ and see if it's working as expected now?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/3361#issuecomment-290764702, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1EEkSXiXtisCahbaW-lIXv-jW8LqdMks5rrSy_gaJpZM4LUd48 .

[aprajita]

Hi @rogerhutchings actually it's not working - it doesn't seem to be taking the tolerance radius in to account. i.e. i can click anywhere in a sim and get a congratulations message. is this still using https://github.com/zooniverse/Panoptes-Front-End/blob/master/app/classifier/metadata-based-feedback.cjsx to test mark success?

[aprajita]

e.g. for image_3 my marker is way off the arc, might be that the pixel scale used in the manifest I supplied is different to the one used to test tolerance

[rogerhutchings]

I've hopefully pushed a fix - can you try now?

[aprajita]

hi @rogerhutchings thanks yes that seems to be OK now - thanks.

now that this works on this test site with these test images, what are the next steps to getting this working on the main SW Demo site?

A coupe of other things that may require slight modifications to the mark testing: the actual data/manifest will be slightly different in that I plan to subdivide the type category to cover all types of training images and will be grouped into those triggering same response i.e. type: SUBJ - ordinary test subject type: LENS subtypes: SIM & KNOWN - this triggers a success message for marks on arcs, and missed message otherwise type: DUD subtypes EMPTY & FP - this triggers a missed message for any marks, and success message for no marks

also you may have noticed x2,y2,tol2 entries in the manifest these account for secondary (and could be more) multiple features that should also be considered as any mark within the tolerance of any of the features should trigger a success message for the LENS type.

many thanks! aprajita.

[aprajita]

Hi @rogerhutchings many thanks for looking into feedback

• shall i send an updated manifest as per my previous message for more testing?

• an important part of SW is the speed of classification so we have selected to "Hide classification summaries" to allow users to go through as many images as possible without having redundant clicks With respect to feedback, I was wondering if there is a conflict with providing feedback when hide classification summaries is on? It looks like the feedback is set to appear on the classification summary page and if the classification summaries are hidden then how can the feedback be shown? e.g. (a) can the feedback if on a classification summary page be shown just for training images not normal test subjects? Or (b) could pop-up message for feedback be a solution, e.g. like in the old SW platform?

Many thanks! Aprajita.

[aprajita]

hi again @rogerhutchings in #3725 the feedback seems to be a fixed message for a given type/success/fail. We've been working on the basis that we could provide feedback per training image in the manifest. Could you clarify that we can still have messages corresponding to a particular subject? It would be good to chat with you about this and what the next steps are for us, please let me know when would be convenient. Thanks & best wishes Aprajita.

[rogerhutchings]

Hi @aprajita. You can have different default success / fail messages for each type, and those can be overridden per subject in the manifest - it's all in the comment, and you can see it on the test project by classifying the Baz subject.

I'm in Oxford on Tuesday / Wednesday, so drop me an email (roger@zooniverse.org) if you're around to talk.

[aprajita]

My apologies for missing that @rogerhutchings

[srallen]

@aprajita I'm going to close this now. As you know, @rogerhutchings has implemented a revised generalized feedback feature and it's best to open specific issues related to that while testing it.

[aprajita]

OK @srallen, i have already opened specific issues for that.

On 29 June 2017 at 21:56, srallen notifications@github.com wrote:

@aprajita https://github.com/aprajita I'm going to close this now. As you know, @rogerhutchings https://github.com/rogerhutchings has implemented a revised generalized feedback feature and it's best to open specific issues related to that while testing it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/3361#issuecomment-312106346, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1EEo2N_JApXLXKTvqtvm126Ex0U844ks5sJA-JgaJpZM4LUd48 .

[srallen]

@aprajita Thanks! Just doing a bit of tidying of the issues assigned to me.