Add generalized feedback functionality for Project Builder projects

[lcjohnso]

A number of projects (Space Warps, Gravity Spy; SW, GSpy) have started testing an experimental feature of PFE that provides volunteers feedback on their submitted annotation at the summary screen. This metadata-based-feedback feature has been hard-coded on a project-specific basis, but generalizing this tool would allow many interested projects/teams to use this helpful feature. This issue expands on ideas discussed in SW-related Issues https://github.com/zooniverse/Panoptes-Front-End/issues/2834 and https://github.com/zooniverse/Panoptes-Front-End/issues/3361, and implemented in PR https://github.com/zooniverse/Panoptes-Front-End/pull/3202. Also, similar feedback has been implemented by Planet Hunters to reveal locations of simulated and known transits.

My interest: I'm developing a number of Project Builder-based star cluster identification projects, similar to the completed Andromeda Project (AP). These projects will use synthetic clusters to characterize the catalog completeness properties (and receiver operating characteristic curve - ROC). We found it important to clarify to the volunteers the purpose of these synthetic objects, so they didn't feel that they were being tested or studied. In addition, volunteers commented that they liked receiving positive feedback when they successfully identified synthetic clusters, reassuring them that they were accomplishing the task successfully. Here, feedback serves a dual purpose: disclosure and training.

Specifications for Feedback: For this use case, it makes sense to include information about the synthetic clusters in the subject metadata, using similar manifest columns as currently implemented for SW:

Type #X #Y #Tol

SIM 100.5 200.5 50.0

When an image with a synthetic cluster is shown, the following would be the desired behavior for the feedback:

If volunteer marks a cluster (not galaxy or image artifact marker types) within #Tol pixels of synthetic's (#X, #Y) input position: 1) Display circular marker at (#X, #Y) with radius #Tol. 2) Display text either in text bubble next to synthetic object marker, or as part of the summary screen text on the right side of the UI. Here, the subjectSuccessMessage could be the same text every time, defined in workflow_contents: "Congratulations, you found a synthetic cluster!"

Optional Features:

• The subjectSuccessMessage text could rotate through a small number of randomly chosen messages to mix things up, or in the future could provide tailored feedback based on the specifics of the individual synthetic. This latter enhancement could be implemented using a #SuccessMessageIndex column that refers back to an array of responses that could be stored in workflow_contents.

Example: Here's what the feedback looked like for AP:

Differences from current SW implementation:

• No DUDs for this use case, only SIMs.
• No negative feedback in the case that a synthetic cluster is not identified. Cluster sims will be at/near detection threshold, so unidentified sim is not a failure.

Differences from SW & GSpy implementations:

• It would be very helpful to indicate the location of the synthetic cluster by drawing a marker that displays along with the message, as opposed to simply printing text along with other summary screen content. The image subjects are sufficiently complex that it might be unclear where the synthetic object is in the image.

Potentially Useful References: @srallen's PR https://github.com/zooniverse/Panoptes-Front-End/pull/3202 = implemented experimental metadata-based-feedback. AP's implementation of feedback = L237-L267 in src/scripts/controllers/Classifier.coffee of https://github.com/zooniverse/AndromedaProject/

[lcjohnso]

Tagging potentially interested people: @rogerhutchings, @aprajita, @hughdickinson, @srallen, @mschwamb.

Broad comments about use case specific variations of feedback functionality are welcome!

Also broadly related: other use cases for advice/info/feedback at point of classification submission, as mentioned in Issue https://github.com/zooniverse/Panoptes-Front-End/issues/2814.

[lcjohnso]

Amplifying a point made by @mschwamb on PR https://github.com/zooniverse/Panoptes-Front-End/pull/1934, I wanted to comment on how the feedback use case here differentiates from previous efforts related to Gold Standard feedback (other relevant PRs: https://github.com/zooniverse/Panoptes-Front-End/pull/2923, https://github.com/zooniverse/Panoptes-Front-End/pull/3041, https://github.com/zooniverse/Panoptes-Front-End/pull/3044)

For synthetic clusters, the primary motivation of feedback is disclosure: it is important that volunteers know what objects are synthetic, so they know which identifications are not new discoveries but rather successful tests. The training + positive feedback aspect is also important, but not necessarily the primary motivator. In all cases that synthetic clusters are identified, volunteers should be notified.

In contrast, I argue (and I think Meg would agree) that not all gold standard classifications should be offered as supplemental info to volunteers. For many projects, gold standard data is primarily used for offline data analysis as a means of assessing classification reliability. Therefore, not all available gold standard classifications should be revealed to volunteers, as it might tend to bias or change classification behavior in an unwanted way.

That said: the Milky Way Project's uses gold standard / expert classifications in their "Learning the Ropes" workflow specifically as a training tool. This might be a good example of a happy medium: offering to show gold standard classifications for a restricted subset of training subjects is helpful, but this training-based use of gold standard data should be separate from the team's broader collection of gold standard classifications from experts for analysis purposes.

[trouille]

Note here -- another project that may use the Space Warps style feedback is Edward Gomez of LCO's project. Just an FYI.

[zemogle]

This would be a definite bonus for our project where code identifies NEO candidates but has a high false-positive and fails to find candidates accurately (particularly if image quality is dicey). We'd like exactly the functionality you describe for identifying candidates and for improving our algorithms.

[srallen]

Just a heads up for @rogerhutchings that #3645 is what I ended up doing for SGL, so a simple notification for sim subjects is also another type of messaging that is needed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

Closed by probot-stale due to a lack of recent activity. Please feel free to re-open if you wish to take on this change.