validation: label swap, and activity log (broken data examples)

[dobkeratops]

(EDIT: the existing Explorer view allowed finding these problems. Perhaps just adding additional filters there for time,user would be much better than a seperate log)

i just did a few where i might have done the wrong label (pavement <-> road..) .. but I'm not sure (i had several windows open, had the intent of swapping label (which means i might have searched), and went back to one knowing i'd bee annotating roads but saw 'pavement' in the browser view)

would it be possible to keep track of the annotations a user has done , to allow going back to review such mistakes when fresh in your memory..

Perhaps this could be done with a simple user activity log - image id, label id, .. - then these could be cut-pasted for checks or submitted through scripts before there's a UI for it (the ideal would be a thumbnail of the image and annotation in that log, but just having a list of ID's which could be manually pasted into scripts/endpoints would get the ball rolling)

A browser view showing the images you've been over would be a great thing to have regardless: e.g. being able to explicitely go back and add labels or annotations to images you've been through already, or check what existing annotations they had; plus seeing the work you've done accumulating is a nice motivator <<EDIT: explore with user filter, time option would do that just fine

[dobkeratops]

ok i've been able to find the mistakes in the explorer view, they're on the following images:- (i.e. annotated as "pavement", but should be "road")

"uuid": "fdba4a07-3994-4ec6-9ae8-fdb1abe0e6f6", "uuid": "f983facb-e68f-4ebc-bdd9-bfac648547e4", "uuid": "f6b4a234-f045-4119-bd2e-3a0a29de9705", "uuid": "e5a3f411-cf8c-4fee-b7f8-09742577f2a6", "uuid": "d61bb9ab-5d2e-4baf-a18b-03e971c96455", "uuid": "d61bb9ab-5d2e-4baf-a18b-03e971c96455", "uuid": "baa1950a-8a00-4277-9c3a-1e4f806cfc0a", "uuid": "b7b1cc4a-0ce6-4b8d-9072-19da2be0d6a4", "uuid": "b7721fe4-945f-4064-a46d-b7825b150e72", "uuid": "adcf4051-bf1f-40c2-976b-12715894930a", "uuid": "9a70eed0-100a-4505-9668-962a26646c7e", "uuid": "97e12eff-7cae-4ff3-873f-aca4565ccf3d", "uuid": "8e963248-39b6-4a83-9511-f704af58a2de", "uuid": "8b81a17f-5e39-443c-8a20-2dce7841e9d5", "uuid": "851e4447-a671-4c34-be0a-6bd7f749821d", "uuid": "81e626a5-bf69-4f5c-8845-91c7a2d61649", "uuid": "7e9a583c-d95b-4c03-a252-03075ff464ed", "uuid": "75dce1bb-fc20-44e5-b2ca-a42d354af1a6", "uuid": "70effd8b-552e-4cf9-a0af-06ea165c28e5", "uuid": "6d3d834c-d29f-4673-a5ac-865c176100e1", "uuid": "6e965131-1e3a-4caf-bb76-959df1cd72d7", "uuid": "69ddf96e-3fec-41fc-a42f-cbb34f8a81ac", "uuid": "5b76d28b-dea0-43f1-bafa-4dd2d562ace5", "uuid": "52366fc2-f5ad-42d4-a99e-7f33b93a924d", "uuid": "48a86999-094f-49df-9463-7be0d38552e8", "uuid": "4504e18b-89f6-4f2f-bb5a-1ceccf58d962", "uuid": "3727f9a4-5cc2-4817-ade6-a635c6356986", "uuid": "367c73b4-9713-4b66-9224-f359b8238cb1", "uuid": "166c3e96-a0c3-4ca7-91eb-983ef36670a9", "uuid": "0015729c-b738-427f-b3c6-eebb12c3325a",

• perhaps rather than an activity log, just adding a user/date filter to the Explorer view would be sufficient (seeing how the explorer view let me catch the mistake this time.. but in future these annotations will be harder to search)

Seems I may have made this mistake earlier aswell , as there's some from previous batches I wonder if showing the existing annotations somewhere (faint etc) might help catch this sort of thing, the other thing I was sometimes doing was scrolling the browser view down a bit to see the other end of the image (obscuring the label name in the page..) - perhaps view re-arrangement (toolbar on the left to save space?, trim the black area?) could reduce that hazard

I guess also a future refinement mode could catch any problems like this, ("what kind of pavement .. asphalt, large tiles, cobblestone, ... / not pavement!")

[bbernhard]

edit:

just realized that you are talking about already annotated objects. The below solution only allows you to swap image labels. But I guess we could implement a similar mechanism to correct annotations.

First of all, many thanks for tracking those down!

I thought about manually fixing those in the database, but I gues that's a good oppurtunity to add the possibility to invalidate labels.

Currently, I am planning to do the following:

• I'll give you moderator rights
• With moderator rights you can click on a label with the right mousebutton and remove it
• The label doesn't really get removed...instead we just vote with "no" in the corresponding validation task. There is just one little tweak: As you are a moderator, you have a voting power of 5 here. (i.e your vote counts 5 times as much a a normal vote; the voting power of 5 only applies to this right click menu entry - if you vote in the validation view your vote counts as 1)
• if the probability (probability = num_of_valid_votes / (num_of_valid_votes + num_of_invalid_votes)) that the label is valid drops below 75% we strikethrough the label.
• a label that isn't valid anymore won't show up for annotation.
• the right mousebutton context menu isn't exactly mobile user friendly, but I guess such a specific moderator-only feature probably isn't needed on mobile (if it's needed, we can e.q enable it with long-press)

So the general workflow for replacing a label would be:

• "remove" label (by using your moderator privileges)
• add new label

Here's a short preview:

perhaps rather than an activity log, just adding a user/date filter to the Explorer view would be sufficient (seeing how the explorer view let me catch the mistake this time.. but in future these annotations will be harder to search)

good idea!

[dobkeratops]

right click stuff for developers etc is fine, I think. Even on a touchscreen you can hold for a popup-menu I think.

[dobkeratops]

I just did another mistake, clicked 'done' and realised.. this prompted me to scroll through the road examples to try to find it, I found these incomplete/erroneous road annotations:

"uuid": "24d924a5-8192-4121-a924-901ee250ee5a", "uuid": "555c1858-c674-4fee-be16-973b8f8d4f84", "uuid": "6415af1c-b542-400e-9922-5339ce53f56a",

(a similar sort option would help this scenario .. 'filter my contributions only', 'sort recent first, sort by ..')

the browse/explore view is really useful :)