Pure component and material labels - suggestion

[dobkeratops]

Imagine if it allowed part and material names (wheel head metal wood etc) as standalone labels - without requiring an object :

• if you can browse free labelling you will find some examples I’ve done like this already , to see what it looks like (search for wheel , head , hand ..)

• it decreases the amount of typing to make a label, or list navigation to select a label ( eg currently I think the full list must show every combination of component and object). in the tablet+pen use case (drawing outlines is much faster but typing is a bit slower) this has been extra comfortable. Even when using a keyboard, writing out “wheel of car, headlight of car, car, windscreen of car...” means a lot of repetition of typing car

• it would allow annotating the parts of objects we don’t yet have labels for, eg obscure types of vehicle or device (I’ve seen a cleaning bucket on wheels, “Tricycles” share components with “bicycles”, or there are many types of animal yet to include).

• I would still suggest allowing the specific combinations aswell

• regarding training , there are would be parallels between the component of many distinct objects - eg similarity between “wing of aircraft” and “wing of bird”, “fin of fish”, “fin of rocket” ; whilst I would still say “specific object components” are more important , I think you could still train a detector for standalone part labels

• sometimes those comments are visible in isolation, either detached , or with the rest of the object invisible

• regarding materials, some objects are comprised of multiple materials , eg a tool with a metal blade and wooden handle , a box with a metal frame and wooden panels, a plastic bucket with a metal wire handle ; buildings with stone or wooden walls, and tiles or thatched roofs , etc

• again with materials, sometimes you can see components , bits of debris etc without knowing what they are

Layering could be used to retroactively add the “object name” information , or possibly (harder) a hierarchy feature (like label me) - eg imagine if you could select several polygons and combine them to make the patent shape . I’ll say when annotating with a pen , doing an extra polygon for the whole object is usually fast enough already . I would suggest either (i)trying this at the pixel level (ie “man,woman,car, truck” are distinct image, channels from “wheel, hand, head, headlight” etc - so they all just get painted in allowing overal - and the existing specific part labels just activate both object and part channels) or (ii) assigning an object to the part based on greatest polygon overlap (harder, and you’d probably have to code the pixel overlay first anyway to figure this out). Objects might overlap already so I suspect we’d have to use a many channel image anyway .. there would be various ways to handle this reasonably efficiently)

Some suggestions for part labels: wheel,tire,hub, axel,lid,handle (many hand held objects , also doors),hole, saddle, seat, roof head,tail,neck,arm,leg, foreleg , hindleg (some animals .. “foreleg” is transition between quadruped and biped), wing, fin, flipper (divers and some animals), tailfin ( best name for the rear fins of an aircraft vs calling it tail - tailfin of fish , tailfin of aeroplane ..)

Some labels could be a part or genuinely standalone object: wall

[dobkeratops]

Added information by hand. There were a lot of existing ones I'd done previously, and I wrote a script to find the "orphans" (ones not already in graph nodes), to guide manually plugging the gaps. I think this process works well for growing the label-graph incrementally.

(>), <(<), &(&), "(") and '(')

I thinkso. the arrow -> is the main one.. elsewehre I did a lot of experimental convention ideas like trying to say "X > Y" - his image has more of X than Y .. but those are probably too complicated. In my script i'm just doing this x.replace("-\u0026gt;","->") , which gets the arrows specifically.

The one thing I should say is I'd started using a->b->c as a shortcut for a deeper label path, but there's also a need for making one label pointed at by several others (eg "tiger = carnivore,mammal,quadruped). So far I did those manually (seperate nodes for each link, as per the graph syntax) . I made intermediate nodes that can streamline multiple groupings a bit, e.g. "carniverous_quadrupedal_mammal", etc.

Coming up with an intuitive shortcut for multiple links would streamline it (something like {a,b,c}->d ?), but I'm also worried about making the label parsing too complex (it'll deter other users). I think the simple "a->b" is pretty obvious; many people will have seen graph-viz already. If you wanted many users contributing labels this way... maybe you'd need some dedicated UI (which would verify what users wrote, or ask for specific fields that can then generate nodes) ,or maybe its enough to look at trending labels and add new graph nodes for what we see.

The material/properties system you already have is aimed at the same job, but I do like being able to write these natural phrases in one step (the graph gives us a general purpose way to combine any information in aliases, and record sensible combinations).

[bbernhard]

Added information by hand. There were a lot of existing ones I'd done previously, and I wrote a script to find the "orphans" (ones not already in graph nodes), to guide manually plugging the gaps. I think this process works well for growing the label-graph incrementally.

got it. Your script approach sounds really interesting. I am wondering, if we can provide that somehow as a service? e.g a service that you can query and that tells you which labels are missing in the label graph.

Coming up with an intuitive shortcut for multiple links would streamline it (something like {a,b,c}->d ?), but I'm also worried about making the label parsing too complex

you are right, that's indeed a tricky one. I guess the more complicated the expression gets, the more likely syntax errors will be (e.g imagine missing a , or a bracket). If we wanna go that route, I guess we need - as you already mentioned - some sort of UI help/syntax checking.

[dobkeratops]

Just trying to think what the best option is..

• make the python tool spit out some static pages summarising the graph, listing orphans etc, maybe show an example image per label
• just rewrite the orphan discovery( it’s not much code) in Go and just integrate it into the server
• rewrite it in JS and a serve it as a page , doing the same job

Rewriting the orphan discovery is no big deal. Most of the code was experimental (messing with word vectors which I will ditch)

EDIT: i just discovered all this browsing around your UI and repos, looks handy.. https://github.com/ImageMonkey/imagemonkey-libs/blob/master/python/pyimagemonkey/api.py

I guess I can work from that to try building some of the visualizations I'm after (endpoints to pull in images and annotations.)

[bbernhard]

Just trying to think what the best option is..

* make the python tool spit out some static pages summarising the graph, listing orphans etc, maybe show an example image per label

* just rewrite the orphan discovery( it’s not much code) in Go and just integrate it into the server

* rewrite it in JS and a serve it as a page , doing the same job

Rewriting the orphan discovery is no big deal. Most of the code was experimental (messing with word vectors which I will ditch)

good question. I guess it mostly depends how tightly we want to integrate it into the service. I think such a tool would also make perfectly sense as a standalone tool. The big advantage I would see with that is, that no separation between backend and frontend code is needed - so it could be a GUI/commandline app written in whatever language you prefer, using the ImageMonkey APIs to communicate with the server.

Integrating it into the service would also be possible. But in that case you probably would need to split the code up in a backend code (preferable golang code) and some frontend facing code (html, css, js) - which could be quite a lot of work.

Rewriting the whole thing in JS and hosting it on github pages could also be an interesting option - that way it would be easily accessible by others as well.

i just discovered all this browsing around your UI and repos, looks handy.. https://github.com/ImageMonkey/imagemonkey-libs/blob/master/python/pyimagemonkey/api.py

That's the library I am using to fetch data from the service when training a neural net. :) It's far from feature complete (and probably also not bug free ;-)), but I think it should cover the basic use case:

• search for label expressions (e.g: dog|cat), download the images + annotations

e.g: here is the code that creates mask files with opencv for MaskRCNN.

[bbernhard]

short update:

• I've repaired the broken entries in the database. So the https://api.imagemonkey.io/v1/label/suggestions endpoint should now also return -> instead of ->.
• special characters like -, >, < should now also be allowed in the unified mode.

[dobkeratops]

awesome, that works, we can now add graph node syntax labels in unified

[dobkeratops]

Next feature request - would it be possible to add handling comma separators for multiple label entry to unified? This would be handy to paste a bunch of common labels to save typing (eg grab “street,road,pavement,building,car,person,sky” and scroll through looking for street scenes, paste that into the label and blast through annotating those, and manually add anything else unusual to the scene). This would be a handy stopgap before smart label suggestions (eg driven by co-occurrence)

[bbernhard]

I think that should be possible. I'll have a look at it tomorrow :+1:

[bbernhard]

should be working now :)

[dobkeratops]

Awesome, it works.. that’s a nice bit of streamlining to have

[dobkeratops]

(Just switched back to using this on the desktop.. now you can keep a text window to the side and drag-drop common label groups, it’s an interesting shortcut. I was just thinking how the hotkeys for label toggle would help too .. but overall the iPad is still my favourite device for this)