Feature request - retire mode

[khufkens]

We now have a gold standard and a demo mode.

However, I was wondering if it wouldn't speed up some of the projects if you would have a retire mode. Where if you are a the owner or expert on a project you can prematurely retire or weight a classification more heavily (bounce the classification number artificially to a % of the retire rate) for a subject.

e.g. if I classify a subject the classification count will jump to 8 of 10 classifications needed. Two people then confirm my work (in case I missed something I guess) for a subject to retire. Or set it to 100% so when an 'expert' classifies something the subject drops out.

This would also allow you to for example pay trained people to work on a project in conjunction with citizen scientists. Or speed up things as an expert / owner by classifying with more weight (as you know the data best).

[chrislintott]

At present, we could support this for some projects by allowing project owners to retire subjects via the API. Let us know if you’re interested in operating in such a mode.

Chris

On 6 Jan 2016, at 05:46, Koen Hufkens notifications@github.com wrote:

We now have a gold standard and a demo mode

However, I was wondering if it wouldn't speed up some of the projects if you would have a retire mode Where if you are a the owner or expert on a project you can prematurely retire or weight a classification more heavily (bounce the classification number artificially to a % of the retire rate) for a subject

eg if I classify a subject the classification count will jump to 8 of 10 classifications needed Two people then confirm my work (in case I missed something I guess) for a subject to retire Or set it to 100% so when an 'expert' classifies something the subject drops out

This would also allow you to for example pay trained people to work on a project in conjunction with citizen scientists Or speed up things as an expert / owner by classifying with more weight (as you know the data best)

— Reply to this email directly or view it on GitHub https://github.com/zooniverse/Panoptes-Front-End/issues/2077.

[khufkens]

Yes, I think it would speed things up a lot. If you could enable this, or provide a menu of sorts that would be great. My project is Jungle Rhythms.

This would allow me to work in a dual mode. Me doing a fair bit but with final say and quickly retiring subjects, while contributors go at a slower pace as there should be multiple view per subject.

[chrislintott]

Sorry, I think I'm confused - you want to classify in the interface, but not have your classification immediately retire a subject? Instead, you'd like that to be the equivalent of many other people having a look?

[khufkens]

It doesn't matter to me how it is implemented to be honest. I'm not sure how the retired flags work (if this is a set variable, or if the retired status is only checked based upon the number of subjects e.g. if # subjects > 10 then retired).

In short, I just want to be able to retire subjects quickly if I classify them. And towards the future, give a select number of collaborators the same rights (the latter seems to be the true future feature request).

[chrislintott]

@ggdhines Seems like we could just set a custom aggregation retiring subjects this user classifies?

[ggdhines-zz]

@chrislintott we already have the API ability to retire subjects on demand (I use it for Annotate and Folger). @khufkens you can download a dump your classifications and do your own analysis to determine which subjects you want to retire. I can share the API code with you but I just realized that the dump won't include anything that allows you to identify the users (e.g. email address or user id or ip address), each user is only identified by a number. I think changing that would would take a lot of effort and break several things, in particular our policy about the information we make public about users. A custom aggregation job would not be trivial. I don't have the time and don't want to set a precedent.

[khufkens]

@ggdhines How does a dump help me with retiring subjects, I can't follow your train of thought. I'm looking for a solution that would help progressively, not retro-actively. I also thought that users can be identified by their zooniverse account handle (user name).

[ggdhines-zz]

The dump gives you all of the classifications for your project to date. So to do the analysis you need (in order to determine which subjects you want to retire), you need to get the dump. Users are identified by a username but that username is not included in the classification dump - instead we give each user a unique id number.

[khufkens]

But there is no way to work more progressively, directly from the interface? If a subject is registered as retired the moment I submit it, it will not be provided again to other users. This is not the case if I work retroactively, potentially wasting user effort as they will view a subject which will be retired anyway (the moment I run the API hooks).

[ggdhines-zz]

Other than making an API call, the only way to retire a subject is via the classification count. So classification count is the only way to retire a subject "progressively". (Setting the classification count threshold is done in the project builder.) There's actually not much risk of wasting user effort with a "retroactive retirement". Suppose you have 20,000 subjects (I think this a reasonable guess for how many JR has) and get between 1,000 and 2,000 classifications per day (just checked your classifications and this again a reasonable number). Then the expected number of classifications per day on each subject is at most 0.2. So if you (as an expert) classified a subject which should then be retired and you ran a "retroactive retirement" scheme once a day, then the expected wasted user effort is very small.

[khufkens]

True, it just feels cleaner the other way around. Does the platform allow me to pull the whole database every single day and automatically (programmatically)? Under the data exports tap in the project builder it says only daily exports are allowed, and not guaranteed. I would be happy to write some code to retire those subjects this way, but doing it manually is not an option.

[ggdhines-zz]

@khufkens we don't currently have a system set up to automatically download classifications. I believe this is a long term goal but don't know the timeline (and don't think its any time soon). Just to be clear - even with such a system, you wouldn't be able to identify specific users in the classifications.

[khufkens]

@ggdhines I'm confused, so how were these plots created? https://seasonspotter.wordpress.com/2015/11/30/season-spotters-unregistered-volunteers/ Not sure how this would be possible if you can't count the contributions of people.

[ggdhines-zz]

Yup - you're right. The user ids are available.

[khufkens]

So this would more or less solve the issue? (If I can query my contributions and selectively retire them using an API hook, a posteriori)

[marten]

I've only just noticed this thread, but we do already have a realtime server capable of handling/implementing custom retirement rules. Simply weighing some users' contributions higher would seem to fit neatly within the scope of things we want that code to do. We could set that up to work for your project, although we'd have to discuss how you envision us knowing which users should be weighted differently.

[khufkens]

@marten This would mainly concern me as an owner and some people who are really familiar with the data (I'll ask for their user name / handle).

[marten]

Cool, if it's only a few people and it doesn't change frequently, I think we'll just put it in a config file.

[khufkens]

If you could add me for starters that would be great. There might be two other people to be added later, but I'll ask if they want the responsibility first (it ups the stress level of an otherwise quite relaxing project a bit).

[srallen]

Closing since this is more a back-end feature request and something that may now be possible with custom rules implemented in Caesar. UI is planned for the project builder lab.