Same subjects served repeatedly to some users

[VVH]

One of our moderators says they and other users are served the same pages repeatedly. Clicking 'I'm done' and 'There's more to transcribe' doesn't prevent them from seeing those images again. The only way to stop seeing them is to add one line of transcription and then move on. Our mod writes: 'There must be a couple of thousand pages in the letters category that I have never seen (and it's unlikely that they've all been retired), yet as I have explained before, I keep being shown the same 2-3 dozen images over and over and over again in an endless loop over hundreds of reloads. It's maddening.😦 And it's not just coincidence, because it's been like that for months now. Only when I do a partial transcription of one of the looped images will it be replaced by a new one - which proves that there are still plenty of images out there that I haven't seen yet.' While I think that uploading new data will solve this problem, it strikes me that the site is not working as it should. Does anyone have any ideas of why this might be? Has it come up on other projects across the platform? @camallen @rogerhutchings?

[camallen]

I thought clicking "I'm done" submits a classification for that subject, is that right @simoneduca ? If not then the user has not technically seen the subject so it's available to be shown to the user again, this is by design.

Reloading the page will show the same subject until the user submits a classification for it. This is by design so moving to a new subject will depend on the "i'm done" behaviour above.

@VVH I can look into the available pool of subjects for that user, i'll need a username though.

This does sounds like a bug of some description, it'd be nice to debug this with more information.

[VVH]

Username @mutabilitie. It would be best if hitting I'm done meant a user was never shown the same subject again. That was one goal of the I'm done button.

[simoneduca]

Just to confirm: clicking I'm done and then selecting "No, there are still things to transcribe" submits a classification.

[simoneduca]

Same for "Yes, everything is transcribed".

[simoneduca]

@camallen have you found anything?

[camallen]

There are lots of unseen images for that user on the letters set (2778). I've tested the selector returns unique data across the set as well. So if the client is requesting new data from the API via the normal mechanism then why is the client stuck showing the same image over and over again?

I'm guessing this is a client issue, old code perhaps? Perhaps getting the user to clear local storage? Is there a way the code can clear this data on somehow?

[simoneduca]

annotations storage is cleared when the queue advances and the subjects storage gets updated with the new subject data. I'd be helpful to see what's in her local storage.

[VVH]

Mutabilitie reports back that I slightly misrepresented the situation, so here's a direct quote:

"Basically, whenever I'm being shown a document that I don't fancy at the moment, I reload https://www.shakespearesworld.org/#!/transcribe/2778, which will pull a new document from the pool (I prefer that to just clicking "submit" right away, because a lot of the time I'll be happy to transcribe that page when it comes up again at a later point). Until a few months ago, this worked fine, because there were lots of documents in the pool and I would easily find a new one that I preferred after reloading two or three times. Now, though, the entire pool seems to consist of about 2-3 dozen images, which I have already seen lots of times. This means that when I reload https://www.shakespearesworld.org/#!/transcribe/2778, I will keep being shown the same 2-3 dozen images, until I flush them out of the pool, as it were by clicking "submit" (with or without transcribing a token like). I'm positive that this wasn't an issue before."

Sample subject images at issue include 53901, 53894, and 53711. These are just a few examples.

[VVH]

Hey @simoneduca I know you have a lot on your plate, but I wonder if you or @rogerhutchings have any further thoughts on this?

[camallen]

@simoneduca could we modify the site to build in some debugging to gain visibility to the selected subjects, especially on a browser refresh event. I've got some thoughts about how we do this.

We could deploy a staging / preview site that posts these custom messages to a known server to avoid polluting the main app. Longer term we could build it into the client and look at enabling the behaviour via query params / feature flags, etc.

[simoneduca]

Definitely something worth testing out.

[VVH]

For reference @ mutabilitie is seeing subject 116176 repeatedly, even after submitting a few lines of transcription.

[VVH]

Interestingly, I just saw this similar looking issue over on AnnoTate, same codebase same old problem? https://github.com/zooniverse/AnnoTate/issues/203

[VVH]

Another case of the same subject being served twice to the same user at different points in time. https://www.zooniverse.org/projects/zooniverse/shakespeares-world/talk/192/426311?comment=819713&page=1

[CharlesDorsett]

I don't think it makes any difference whether you click "everything is transcribed" or "no, there are things left to transcribe." For the past 8-12 months, I have almost always managed to transcribe everything, and I am fairly certain that the first time I did 1277450, I got everything. So the issue is not that people said the doc still had untranscribed stuff in it.

I do think we need to keep from confusing this issue with the "feature" of returning to the same document repeatedly until you click "done". I like that feature! It keeps me from having to leave my browser open. The buttons could have better labels, though :)

[VVH]

Hopefully helpful information for you @camallen. Thanks @CharlesDorsett

[camallen]

@CharlesDorsett thanks for that information, did you get the subejct_id 1277450 from the network traffic tab in the browser or some other way? I'm wondering if you could provide the console log / network traffic for subject selection to see how this error state is happening.

No problem if not, the fact you had the subject id led me to think you may have some skills we can use to get more information on this issue.

[CharlesDorsett]

@camallen , I got the number from the email notification that I receive for any additions to the talk page on Shakespeare's World. I have skills, sure, but I don't think I'm in much of a position to troubleshoot this for you. I can access that talk page again, but what would you want me to get for you from there?

[camallen]

Excellent that makes sense re talk. I just need to get a user that can reproduce this behaviour so i can inspect the state client & server side. I've got leads with your and mutabilitie's logins so i'll just work with that on my end. Thanks for the prompt response, it's appreciated.

[camallen]

I've just run local versions of the front end + back end together masquerading as Mutabilitie and failed to get stuck on a single subject.

We did however find a seemingly unrelated bug that may have side effects related to this issue. We've released a fix this morning. @CharlesDorsett please reload the page to get the latest code and let us know if this issue persists. If the issue does persist i'll try another avenue that could be causing issues.

@VVH could you please report what i said above to Mutabilitie?

[CharlesDorsett]

I will give it the old college try. But just so you know, I've only noticed the duplication in 3 of the 575 pages I've worked on. After a while, it's hard to recognize whether I've seen a particular page before. I will be sure to let you know if it happens again.

How can I find out the version of the page I'm working with?

[camallen]

Thanks @CharlesDorsett, all understandable and we really appreciate your help with this.

[VVH]

Yes of course @camallen, thanks.

[CharlesDorsett]

No problem, @camallen - it's fun!

[simoneduca]

Fixed via https://github.com/zooniverse/shakespeares_world/pull/356