Seen before & retired image flags

[camallen]

I've got an image that i've seen and is retired but it only shows 1 flag (already seen) at a time.

[edpaget]

Do we need to have both flags?

[camallen]

I think so, we could just say retired and already seen.

[chrissnyder]

Is there any circumstance in which a user would be served a retired subject when there is still data that needs classifying on a project? If not, then the retired flag signals something else entirely, in that the project's current dataset is "done", which is a different message than "we just gave you a single subject that is done".

[chrislintott]

3The only example I can think of is if an expert user was working through a dataset that had already been classified by volunteers in order to augment the gold standard dataset

Cheers

Chris

On 17 Jul 2015, at 14:47, Chris Snyder notifications@github.com wrote:

Is there any circumstance in which a user would be served a retired subject when there is still data that needs classifying on a project? If not, then the retired flag signals something else entirely, in that the project's current dataset is "done", which is a different message than "we just gave you a single subject that is done".

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/1050#issuecomment-122278798.

[mrniaboc]

I think it would be cool to redirect people at this point, instead of just telling them they've seen the image before, we could give them a message like "Well done! It looks like you've seen all of the images on this project. We could really do with your help over at "

[vrooje]

Agreed, an optional redirect is a good idea and it's a good opportunity to try some experiments too.

However, we need to make sure zooniverse/Panoptes#1069 is solved first.

[mschwamb]

Wanted to chime in and say that the out of data message like @mrniaboc suggests has been helpful for me in the past especially before the stats page for Ouroboros projects. When several people saying on Talk they were getting that message, it was the sign that we could push live the next set of subjects that were prepped and waiting.

[mschwamb]

This came up in a talk thread today on Planet Four: Terrains. I noticed two users are very similar in posting and sign with the same name, Well it happens to be a super classifier who is very active on Talk hit all the available data (we have something like 8000 subjects) and kept getting the already seen flag on Planet Four: Terrains and didn't know what that meant apparently so made another account to start classifying again without the message and posting on Talk. This is absolutely an edge case, but I think it points to classifiers don't understand what the 'already seen' means.

[camallen]

@mschwamb i don't consider what that user did as normal...why would they create another account to do the whole set of data again? Did they not think that they would repeat their previous work?

I agree the messaging in the UI can be improved. It'd be nice to run so a/b splits on this to see what works best.

[mschwamb]

@camallen - totally agreed it's an edge case. Never had this happen before as far as I know. The reasoning I got so far is they partly thought it was either glitch since images kept coming and they felt that that with a new account they'd see different images. They didn't quite get from the orange banner that there was no more data. And since the banner went away with a new account, they decided to keep going.

[chrissnyder]

Not sure where to go here, would love more input.

The current messaging is clearly insufficient and unclear if a volunteer didn't make the leap between "you've seen this before" and "you've seen everything there is to see".

Maybe we need a much more explicit banner across the top of the classifier stating something to the effect of "hey, thanks so much, you've seen and classified everything there is to see on this project! go to Talk, etc".

Thoughts?

[mschwamb]

That would be fantastic @chrissnyder . I think it should be a popup like the reminder to sign in.

[aliburchard]

like @chrissnyder's suggestion. I'm partial to having a banner across the top of the interface itself (not on the image itself) that says something along the lines of "all the images for this project have been classified! we'll let you know when we add more data, but for now, please go here"

[mschwamb]

Related to this thread - there's someone on Comet Hunters Talk asking why they're seeing the 'already seen' label all the time now. I think that is another example that people don't really understand the labels going on the images, and a pop up message saying 'thanks you've completed the data' would be a better route, when there is still data live and can be classified but the particular user has seen all of it.

[chrislintott]

I agree with this. @brian-c?

On 21 Feb 2016, at 07:25, Meg Schwamb notifications@github.com wrote:

Related to this thread - there's someone on Comet Hunters Talk https://www.zooniverse.org/projects/mschwamb/comet-hunters/talk/210/40383?page=1&scrollToLastComment=true asking why they're seeing the already seen all the time now. I think that is another example that people don't really understand the labels going on the images, and a pop up message saying thanks you've completed the data would be a better route.

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/1050#issuecomment-186766517.

[yshish]

I don't understand your reasons for using a banner instead of making impossible to classify the image again. I'm just afraid that as long as there are images which have been seen already possible to be classified, there always will be volunteers classifying them. (For example kids from my class who almost don't speak English but love easy projects like Snapshot at Sea where you have a simple task with yes/no answer. I'm sure no banner would stop them from clicking as long as they can click on the button.)

[vrooje]

I think you've hit on exactly why! It's important for many projects to be able to stay open so that educators can use them, but of course it's also important to be clear in that case that additional classifications aren't contributing as much to the science. The "Already Seen" banner isn't clear enough, but I don't think we want to completely prevent classifications, either...

[yshish]

Thanks for explaining, Brooke. It makes more sense now, although I still don't see the current banner as a good solution. Maybe adding a pop-up explaining the situation would be more helpful?

[camallen]

Some volunteers are having a similar discussion on talk, https://www.zooniverse.org/talk/17/43234?comment=86295

I agree the current banners aren't messaging the intention well enough. In my opinion we should be signalling the following to the user:

1. The subject is retired but they haven't seen it
   • E.g. when the user has seen all the non-retired data
2. The subject has been seen by the user before
   • User has seen all the data in the workflow (power users)
3. The workflow subjects are all retired
   • Currently fires the we're done now banner.

I think I've covered all our bases above but add more if needed.

All points above should note that new effort may not contribute more information to a project. We currently only provide the seen_before, retired and workflow finished flag to the UI but should probably also provide the user has_finished workflow field as well.

[mschwamb]

@camallen I'm not sure people will understand 'retired but they haven't seen it' messaging. I think it will be the same issues as the current 'retired tag'. Is getting classifications on retired subjects that a person hasn't seen actually useful and productive use of their time or a significant gain for the project? I'm not sure. My suggestion would be to direct the person to Talk or another project until new data is loaded onto the project with a message box if a person has seen all the unretired subjects on the site, and if they want to ignore that message the retired/already seen banner can show on the subject as well

[camallen]

@mschwamb I was talking in general terms and not a specific text based message. As per the link i provided, perhaps a visual UI component / iconography instead of text?

Agree we could do better than just putting a label on an image and instead offer alternatives to classifications.

[mschwamb]

@camallen I think I'd have icons and text, but icons on the project cards on the project list might be useful for people perusing for projects to identify which ones need to help and which ones are complete/out of live data

[alexbfree]

icons on the project cards on the project list ... to identify which ones need to help and which ones are complete/out of live data

I really like that idea!

[mschwamb]

The situation came up again. The volunteer who was helping sort the Already Seen classifications now just finished all the live data on Comet Hunters and didn't know that. This situation again highlights that people don't understand what the Already Seen and Retired tags means and it's not signalling that they've finished. It's jargon that people aren't understanding.

I hope a better solution can be implemented soon. A pop up message saying 'thanks you've completed the data' would be a better route, when there is still data live and can be classified but the particular user has seen all of it. Any thoughts @chrislintott and @brian-c ?

[aliburchard]

+100. The need for something besides these image-banners has been a universally acknowledged issue for months now, and is driving many volunteers and project owners mad. Why not just use the already implemented "this project is out of data" banner that appears at the site-level instead of a banner on every image?

[brian-c]

Is the actionable thing here to remove the "Already Seen" and "Retired" banners, and change the "Looks like this project is out of data at the moment" messaging to "This project has run out of new data for you"?

[vrooje]

Yes, I'd be in favor of 2 banners (by which I mean, big messages across the top of the whole classification interface):

(All subjects in active workflows are retired) "All the data in this project has been fully classified (at least for now)!"

or, if that's not the case but a user has classified all the non-retired images: "Looks like you've classified all the active data on this project; thank you!"

Each of those could then have a "Join the conversation on Talk, or try out one of our other projects" redirection...

On a related note, is there anything in this process that could notify the project owner that either of these 2 things has happened?

[mschwamb]

@brian-c I think it's slightly different from what you propose. I'd keep the 'Looks like this project is out of data at the moment' banner that pops up when the project is out of data .That one is working now and gets the point across for most people when that situation happens,

But for those people who've classified all the live data but the project still has subjects in need of classifications, those volunteers don't understand what the 'Already Seen' and 'Retired/Finished' Banners across the subject image mean. They think they're still being useful if they classify it again or classify a retired image. They don't understand the terminology/jargon. So it would be adding in a banner message saying ' You've classified all the available live data on this project. Thanks. Check back soon for new data' when this situation hits.

For now you might want to also keep the 'already seen' and 'retired' orange flags as well since there's still a bug on the backend sending people duplicate images to classify and that might help with debugging that cc @camallen

Agreeing with @vrooje my past projects have gotten in the past a good bump by the redirection link after another project is out of data and the message went up suggesting Planet Four or Planet Hunters, so having a automatic redirection to another similarish project or even random live project in the out of data banners/message would really make use of the strength of being a Zooniverse approved project and the power of the platform. But I see this as an enhancement

[camallen]

can we just skip the already seen images in the UI? If we're showing them to users but not messaging this then i'm worried we'll lose the reporting element (currently by volunteers) when this happens (and it shouldn't).

[camallen]

the api needs to add a response about power users that have seen all the available (non-retired) data still, https://github.com/zooniverse/Panoptes/issues/1807

[camallen]

We're going to deploy the api feature to flag when the user has seen all the subjects today (thanks to @zwolf), we can use this extra info to skip already seen / retired images until the user has seen all the data or the workflow is finished (at this point we should message that they should try another workflow / project). Perhaps somewhere here we could filter the incoming data using the new state.

I think if we "clean" the classify subjects and message properly with the new state, we'll solve @mschwamb (& others) concerns above about messaging. I still think this needs more work in the UI (consistent icons?) but this should get us started.

[camallen]

https://www.zooniverse.org/projects/birgus2/western-shield-camera-watch/talk/372/62840?comment=151724

just noting how confusing the current banners are and that this part of the UX needs some love.

[mschwamb]

These banners are still confusing people, they don't understand what it means . See https://www.zooniverse.org/projects/mschwamb/planet-four-ridges/talk/667/171403

[srallen]

They've been causing confusion on Gravity Spy too. Most seem to think 'Already Seen' is a bug.

[srallen]

Since we've been doing maintenance work on the classifier, I found in the code that the banners are clickable and have an explanation, but the fact those banners are clickable is obviously not being discovered.

I'm thinking about removing these pop-up messages since I don't think they're being used and where they are in the code base doesn't make sense. We need a better solution (if that was an attempt at one).

[camallen]

@srallen i've been wondering if the front end can traverse the returned set of subjects and skip already seen subjects returned from the api? If there are none then revert back to the first seen / request a new set (let the api catch up).

Also the api does serialize if the user has finished the workflow from a subjects?queue=true api call, which can be used to determine any skipping feature. I.e. if finished then don't skip just show them something.

[mschwamb]

I never thought to click on the banner so never saw the popups. . I actually like idea of a pop up/ or message that shows like when the project is fully out of data. If something like that just showed up and had a link to the other workflow if active there is one or another project in the same field/category. One of the nice features of the old platform was that when a project was completed or a workflow completed there was a message put up to direct people to another project, which really helped. When MWP completed the note said to go to Planet Four and we got lots of classifications. With the current already seen we're not leveraging the power of the platform directly people to projects that need classifications.

[yshish]

What about combining the Already seen banner with a pop-up message? Like for the first time a banner appears to a volunteer, there would be also a pop-up explaining what that banner means (and it could also inform that they can click on the banner anytime in future to see what individual banner means). Such a pop-up would only display for the first time the volunteer gets it, and again for a banner of a different type (there are Already seen and Finished). I think this could be less confusing than the current only banner situation and it wouldn't be annoying as the pop-up would appear only for the first time.

And a pop-up linking to other project would be also fine. It could be displayed somewhere above/below the picture in the classification permanently - once the banner appears.

[srallen]

When a project is complete, a separate banner does appear at the top of the project with a message directing them to the results page. It could be updated to also include a link to the projects page:

I think a combination of filtering and messaging will need to happen because sometimes the 'already seen' banner for a particular user is only present for a very short time since some projects upload new subjects nightly or weekly.

I can take a look into having that pop up, which is functioning as a tool tip, open as default when the banner first appears.

[mschwamb]

@srallen

I think a combination of filtering and messaging will need to happen because sometimes the 'already seen' banner for a particular user is only present for a very short time since some projects upload new subjects nightly or weekly.

Agreed. I think you could leave the orange 'Already Seen' label on the image and add a pop up as you suggested. I agree about the text/combining of messaging. I can help /draft with the wording for this new pop up if that would be useful

[mschwamb]

I think this sums up the current understanding of the 'Already Seen' banner:

I keep getting already seen pictures. So what do I do ? reclassify them ? is there a way to click them away and get new pictures ? will listening to David bowie's ziggy stardust influence the result ?

https://www.zooniverse.org/projects/mschwamb/planet-four-ridges/talk/667/174304?comment=303974

[camallen]

@mschwamb there have been some issues with one of our selection services (one ridges is using). We're working on it and hope to have better news soon. In the meantime you should tell that user that listening to Bowie never hurt anyone ;)

[mschwamb]

@camallen thanks for letting me know. I think these people are close to being done with the live subjects given that they appear to be super users. I think the bigger problem is that this messaging isn't clear. It seems like it wouldn't be too difficult to add something similar to when there is no data on a live site kind of message in a box to the workflow page when a person gets an already seen.

[mschwamb]

Volunteers still don't get these labels. Every time we're close to finishing a dataset, I get questions about these. Many people don't get this means they've seen all the available data on the site but there is still images that other people haven't classified and are not retired. Any chance there can be better messaging?

[vrooje]

Hi all, in the PRN Irma project we are experiencing a phase at the end of each new data release where all volunteers get are the Finished/Already Seen banners and they don't get new subjects (they just keep drawing from the same batch of ~10) if they refresh. But the "out of data" message is not showing up. I'm getting questions as to why this is happening and honestly I'm not sure? The stats page shows that there are about 50 unretired subjects in 1 workflow, but I don't understand why I'm not getting them on the classify page when I refresh.

[vrooje]

Hang on, I think I know why. It's because every subject in the default workflow was finished, but there was another active workflow that wasn't finished, and the classifier wasn't switching to that because it was trying to load the default workflow. When I removed the default flag from that workflow, it switched over to the other one.

Is this intended behavior? If so I don't think it's messaged well to project owners as a potential consequence of selecting a default workflow.

[camallen]

@vrooje yeah - i think this is due to the fact the UI wires up the default workflow on page load so selection will happen on that workflow. You have to switch the default workflow over so that new users get the UI wired up on that.

Existing users that haven't reloaded the page won't get the new workflow, instead the UI will keep requesting data for the default worfklow when the loaded the page. Having the ability to message these people without resorting to finished / seen flags is something we should look to do with a websocket and push from the server.

[srallen]

In addition to activeness, I could have the front end check for completeness too and then fall back to a random selection of the active workflows if the workflow selected is complete.

[camallen]

Sounds ideal!

[simensta]

The Already Seen/Finished flags are causing some confusion in the Decoding the Civil War project, as well. See this Talk discussion for the details.

Is there a way to communicate that a volunteer has seen all of the un-retired subjects for a project?