brainstorming - 'out of context' training

[dobkeratops]

not a suggestion for the site, just a NN training idea..

I was just thinking about how you could train a neural net to get a general sense for scenes; I notice in LabelMe there's some images named 'out of context' with what appear to be photoshopped nonsensical situations (e.g. people standing in impossible places, etc etc). I'm guessing thats what those were for.

Given the 'alpha-channel'/'outline' data (i.e. your smart-annotations or any other traced boundaries), I wonder if these 'out-of-context' images could be generated randomly, giving you an endless training set , and you train a net to determine if an image has such out-of-context pastes or not - although you would need images with the originals to ensure it's not just recognising the set of pastable objects, maybe it would also take something more elaborate like adjusting it's brightness to match the surroundings, since many 'cut-pastes' would be obvious from lighting.

What I'd really be hoping for is the earlier layers of such a net would develop a sense of scale and surfaces (which surfaces support which type of objects). (The same early layers could be forced to still recognise the pasted objects for what they are, again to give more training examples?)

In particular I'd hope a net could learn a bit about drivable and walkable areas by learning when a person is on the road ,or when a car is on the pavement

Alpha channel outlines I'd been doing manually (you could generate those from any polygonal or smart-annotations...)

.. pasted over real scenes..

Perhaps it would be possible to do something 'GANN'-like ... train 2 nets adversarially to figure out viable locations given real images and a set of 'sprites' extracted from them (the second net is rewarded if it manages to fool the first net that it placed something in a way that looked 'in-context'..)

[bbernhard]

Hahaha...that's a nice idea - I really like that. In particular, the idea to auto-generate those images is pretty cool! I wonder if that could be a candidate for gamification (e.q guess if the image is fake or real) or if it takes too long until that gets challenging.

[dobkeratops]

I wonder if there is also utility in asking it to recognise the pasted car as a car (whether or not it's in context) - as that might help the net recognise the car more independently of the background (I think with the less clear examples however, the background context helps it

If given scenes with further masks painted (e.g. some indication of layering) it might be possible to help it recognise how things are occluded , by pasting behind things - train a net to guess where the rest of the car is

furthermore given 'pavement' and 'road' channels, you could accelerate the process by using those as guide to place the pastes (people on pavements, cars on the road, then the contextual training can get straight onto learning about scale and direction)

[bbernhard]

furthermore given 'pavement' and 'road' channels, you could accelerate the process by using those as guide to place the pastes (people on pavements, cars on the road, then the contextual training can get straight onto learning about scale and direction)

good idea! I guess that could be something worth to invest a bit of time to. Not sure though if it's something that should be part of the imagemonkey-core package, but I guess it could fit nicely into the imagemonkey-playground. :)