Improve "make labels productive" workflow

[bbernhard]

As there number of labels is growing enormously lately (many many thanks, @dobkeratops), I think we need to change the way how labels are added ("made productive").

I definitely want to make free labeling happen in the near future, but as this requires some pretty big structural changes in the backend (in software terms I would say it's like going from v1 to v2) I am not sure really sure how long this transition will take (my gut feeling is, that this will break quite a lot of integration tests, make existing workflows obsolete...etc).

The problem with those big changes is, that they are pretty time consuming - at some point I'll probably create a new branch (v2) where most of the new development will happen. But that also means, that there won't be any big changes on the v1 version anymore (except for bug fixes of course).

But before I completely focus on the v2 development, I would like to get a few "easy wins" done first - I think there are quite a few incremental changes that could really improve the overall experience and make the time until v2 is ready more pleasant.

Another big topic that I would like to tackle before v2 is: "automate the boring/time consuming stuff". At the moment, there are quite a few things that are only partly automated and which require my constant attention. e.g:

• the whole process of making a label productive
• monthly backups to archive.org
• weekly server backups
• keeping the servers up-to-date and patched
• keeping the third party dependencies (i.e libraries) updated and patched
• ...

Which also brings me to the topic of this thread here. Namely: How to improve the "make labels productive" workflow.

At the moment, there are quite a few steps involved until a new label is productive (i.e available for annotation):

) first the label needs to be added to one of the `.jsonnetfiles. (https://github.com/bbernhard/imagemonkey-core/tree/master/wordlists/en). Scene labels will be added to themetalabels.jsonnetorcountries.libsonnetfile and normal labels either to thelabels.jsonnet, oranimal.libsonnetfile. For each new label that will be added to those files, a UUID needs to be generated (at the moment, this is done manually by copy/pasting the output ofselect * from uuid_generate_v4();`)

*) the changes are then committed to the github repo. travis ci will detect that there's a new commit and runs some unit-/integration tests (e.g it checks whether the UUIDs accross all jsonnet files are unique, if there are no label collisions etc...)

*) if the continous integration tests show all green, the files are then manually copied (via good old scp) to the server

*) next, a database backup will be created - also manually

*) the new labels are added by invoking the populate_labels binary - another step that's done manually

*) existing (trending) labels will be transitioned by executing the make_labels_productive binary

*) finally we are done

So there are quite a lot of manual steps involved, until a (trending) label will appear as productive on the server. As this whole process requires a lot of attention and manual input from my side, I would really like to improve that.

Personally I would like the achieve the following:

) there should be an easy way how users/moderators/admins can request new labels. ) the requested labels should be automatically (with no user input) be pushed to production (ideally after the CI gave green light) *) the whole process should be much faster than the current worflow (i.e the labels should be available latest a few hours after requesting them).

At the moment I am not really sure how the whole thing could look like in practice. Maybe we add another tab to the profile where one can request labels? (maybe with a checkbox that marks whether it is a normal label, a scene label and/or a part of something (e.g wheel/car))

I know, this sounds a bit strange - having another page where one can request labels although we already have the labeling mode. But I think this would be a rather small change (as we do not have to touch much of the existing functionality) that could result in a (big) improvement.

Any opinions/suggestions are really welcome :)

[dobkeratops]

Dedicated UI for label requests is an interesting idea; I was going to suggest flagging particular images as “requests by example” - but a seperate UI could also request labels for which you don’t yet have images. (This can also serve as a request for images)

I might worry a bit about spreading yourself too thin though, eg could you stick with a forum procedure plus example images? .. something like the RFC processes some projects have..l this would let you focus on refining the tools and browse modes. What I imagine is handling labels in batches (eg many related labels for a group of images, or a domain like ‘fruit’ being blitzed in bulk), rather than adding them piecemeal.

You could probably consider this in conjunction with the natural language processing implications ... I see 2 ways this could be done - separating the label list from a graph that organises them, and allowing a word embedding to be used ... word embedding handles the ‘fuzziness’ and group similarity (things like canine, dog,pooch being slightly different points but close together, such that a search for any one can yield related items sorted by similarity)

I wonder if it is possible to entirely decouple the organization from a plain label list, such that the two issues can be worked on in parallel (trying to partner with a natural language project)

[bbernhard]

Interesting thoughts!

Hmm..what if we use github?

I mean, we already have the trending-labels repository (https://github.com/bbernhard/imagemonkey-trending-labels) where new labels are tracked (at the moment a label needs to used at least 20 times, before it will appear in the trending labels repository; but that threshold is configureable..so we could even set it to "1" which would result in a ticket for every newly added label)

In order to mark labels as scene labels we could use github labels/comments (e.g one could either add a comment "scene label" or tag the github issue with a custom "scene label" github tag)

In case the label is misspelled, one could add a comment to the appropriate github ticket that says "rename to: dumbbell".

If the label should be made productive, add the tag/comment "approved" to the appropriate github ticket.

In the background, a bot periodically scans for approved labels, adds them to the appropriate *.jsonnet files, kicks off a new CI run and pushes the label to production.

I think that could work pretty well. I guess we could even improve the bot and e.g add an "synonym detector" later on. e.g: someone adds a new label "coast". The bot could now create a github ticket with the something like this:

**coast** was added.
number of occurences: 25

similar labels that are already in the dataset:
**coastline**

You can now reply with an "approved" in case you want to push the label coast to production or with rename to: coastline in case you are fine with the already existing coastline label.

So it's a bit similar to a chat bot. You tell the bot what to do and it will do it for you.

[dobkeratops]

Ahh “chatbt” approach.. semi automated. Could be interesting mid ground between something free form and formal. The advantage of github (or similar) is

• easier to collaborate with others on the label list ‘task’
• ability to add links for reference
• ability to hold discussions

But in an ideal world where you have devtime for every feature, everything fully integrated on the site would be nice

Could there be a site button .. “request label” which could be a link to a github label request thread?

[dobkeratops]

One more idea regarding free labelling:

Does you system track which user makes each annotation?

If so,imagine if you considered each users free labels as a personal vocabulary, awaiting mapping to the global vocabulary

In most cases you’d hope you can just strip the username; but you’ll have cases where different users choose slightly different nicknames or variations of jargon

Could something like this mesh with handling different languages? It might be useful for distinguishing American vs British English for example ( differences like sidewalk vs pavement, pushbike vs bicycle vs bike, pants vs trousers, etc.. there are cases where a word has an entirely different meaning )

[bbernhard]

Really cool idea, thanks!

That's definitely something we could consider for v2. Haven't thought in detail about it, but I think this even might be a bit less work, than all the other approaches we've discussed before :)

short update: I've started with a small prototype/PoC of the pipeline approach we've discussed above. I've set myself a limit of 2-3 weeks (otherwise I would probably lose myself in details ;)) to experiment a bit with various automation concepts and to see what works best. My goal would be to crank out a working PoC within those ~2 weeks. :)

[dobkeratops]

(just want to link #249 to this discussion .. word embedding - i think these are promising for covering the fuzziness and 'many ways of saying something related' of natural language)

[bbernhard]

short update:

I've been working hard the last few weeks to get a first version of the new "make labels productive workflow" done. Most of the stuff is already working and I am now fixing the last remaining bugs...I hope I can push it to production soon.

Here are a few screenshots to give you a short impression how it will look like:

All the trending labels (i.e basically all the labels that are listed here https://github.com/bbernhard/imagemonkey-trending-labels/issues) will appear as list entries in the following view. (please ignore the garbage names in the screenshot, they are just for testing ;))

When the user clicks the "Add" button, the following popup opens:

You can select whether the label is a normal label (= annotatable) or a meta/scene label (= not annotatable per default). It's possible to add an optional label description in case the label name is ambiguous and you can set the plural form of the label (per default it appends an s to the label name...but you can change that).

After "Done" is clicked, a process in the background creates a new github branch and kicks off a new CI build (to make sure that the newly added label won't break anything).

What's really important for me is, that this is fully transparent (i.e everybody can see what's going on). That's why you can access the CI status and the github integration branch from the UI.

e.g for the trending label abcdefgh (the one you can see in the screenshot) the github integration branch is here: https://github.com/bbernhard/imagemonkey-labels-test/commit/52e41089be25ead0f3f42e5e54ad0abf560f7545 and the CI build here: https://travis-ci.org/bbernhard/imagemonkey-labels-test/builds/569970831

If the CI build runs successfully, the label (the github integration branch) will be merged automatically into the master branch (= label is productive)

@dobkeratops Not sure if it was you, but in case you reported an image today - thanks a lot, I'll take care of that :)

[dobkeratops]

ok could be interesting.. I've got another independent suggestion which could play well with this:

What if there was a label/image explorer with the ability to upvote "the best image for the label" (kind of the inverse of finding the best label for the image). Maybe that could help label curation aswell, (and figuring out visual examples to explain the labels)

just generally, seeing the image that the proposed label is for could help. What i've done sometimes is made several suggestions .. it's always open to debate which one to use.

[bbernhard]

What if there was a label/image explorer with the ability to upvote "the best image for the label" (kind of the inverse of finding the best label for the image).

I really like the idea!

But I guess that could quite tricky once we hit a certain treshold, no? I think it's pretty easy for labels where we only have a few dozen images, but for labels that have a lot of images associated it would be kinda hard, no?

But maybe it would be possible, if we use a numerical scale (e.g 1-10). That way we could sort out "bad" visual examples early and upvote good ones. With enough iteration loops, I think we could end up with good visual examples.

I am wondering if we could get similar results "for free" by just looking at the annotation coverage (but that of course requires that the image is already annotated). e.g: "If the polygon takes up about 2/3 of the image, then it's a good visual example"

[dobkeratops]

similar results "for free" by just looking at the annotation coverage

that could be a good initial guess, and I guess it would usually be good enough there's still going to be the issue of a "whole object" vs zoom in on one part. the zoom-ins on a part would be good examples of the part label of course

My thinking is that this could be a shortcut or bypass for annotation .. a good question for casual users to answer (which word best describes this image , or from these samples which is the best example?)

but for labels that have a lot of images associated it would be kinda hard, no?

• right in these cases you needn't show all of them - you could show 10 at random, and ask which is the best. next time you can show the previous best, plus another 9 random ..