urban ground label ideas

[dobkeratops]

after doing a batch of these, road & pavement certainly cover the majority of cases (so there's plenty to get on with) but a few more will enhance & close some ambiguities:-

hard shoulder (UK english) or possibly shoulder (road) https://en.wikipedia.org/wiki/Shoulder_(road) area alongside a road, only for emergency use, not regular traffic

verge, grass verge https://en.wikipedia.org/wiki/Road_verge

driveway

• the place people drive the car up infront of a house - it's almost part of the pavement, but sometimes an extention of the road.. it's mixed purpose (cars & people)

kerb stone or curb

• the raised edge around the pavements which you can't drive a wheel over

pavement ramp (not sure of the ideal name..)

• opposite of kerb-stone: these are on the edge of pavements as areas for crossing, where wheels (e.g. wheelchairs , bicycles, or delivery-bots :) ) could roll from road to pavement

cyclepath/pavement cyclepath/road cyclepath

• 3 types of cycle path,because they can be either side of the kerb stone, easier to roll into the road, or easier to roll into the pavement, and one has a higher chance of cars going over it, whilst the other is shared with pedestrian space.(or even a completely dedicated area bounded by it's own curbs) This distinction means a single 'cycle path' might not suffice? although if you literally overlap it with road or pavement polygons perhaps that does do enough? What's the easiest way to proceed given how your database works?

parking space

• by the side of the road, sometimes unclear if it's a partition of the road, or a seperate alement car park
• at some point a large area of parking spaces becomes a car park perhaps unify these as one label, maybe spell it out as car_park/parking_space and allow narrowing down later. It might be better to have such a long name rather than leave ambiguity..

central reservation or median strip https://en.wikipedia.org/wiki/Median_strip - the area inside a dual-carriageway. sometimes paved (but not intended to walk on), or sometimes grass traffic island https://en.wikipedia.org/wiki/Traffic_island - common in junctions rename pavement -> paved area? plus add dedicated sidewalk (US english but extra connotation)

• I've marked some of these as pavement, but it's not entirely accurate - it's certainly 'paved'and 'not for cars', but it's often not intended for walking either again it might be nice to have a unified label central_reservation/traffic_island since around complex junctions these do seem to merge. (there are pedestrian crossing points embedded in central reservations)

• The real benefit of this one is simplifying drawing complex roads: with an assumed occlusion rule (central reservations and traffic islands take priority over road annotation) - you could draw a whole dual carriageway as one peice of road, then draw these parts inside them as cutouts (let me try and find a diagram to clarify that). Sometimes this includes grass aswell (which I have definitely not been doing with 'pavement') this could also be used for roundabouts I guess

• changing 'pavement' ->'paved area' and making a new dedicated 'sidewalk' (walkable area beside a road) allows additional annotations (central reservation/traffic island) to combine with 'sidewalk' to make 'paved area'

bus lane

• part of the road really, but often looks a bit different

path

• i've added this label to parks: they sometimes look like roads(asphalt), but they're not intended for cars, they have no markings, and they're thinner. also, a path could be other materials (dirt path, paved path, ...) - but outside of parks i've been marking those as pavement

pavement might want to be renamed 'paved area' with a subtype *sidewalk ('a paved area beside a road') .. but I wouldn't worry about this just yet.. not sure about this.

distinct ground features:-

speed bump

• something you must slow down for, even on a bicycle , sometimes called 'sleeping policeman'

pot hole

• certainly have to watch for these as a cyclist.., even cars have to drive around them someimtes

metal cover in ground .. drain covers or manhole covers (maybe seperate labels) .. i think there's covers for water supply access too (for firefighters?)

building variants -tower -church -house -tower block -office block -skyscraper

I almost get the impression we could have a matrix of surface properties to describe this all, but I think the above distinct labels cover the combinations I can remember..

• is it intended for cars
• is it intended for people
• is it paved
• is it tarmac/asphalt
• is it intended for bicycles
• is it intended for busses
• is it for parking etc..

[bbernhard]

Very very nice!

I am wondering if there is a possibility to make users aware of "scene specific" labels. e.q: If you wouldn't have mentioned pot holes I wouldn't have paid attention to it.

I guess it would be pretty cool if we could detect the scene type(s) and give users some label suggestions ("Most popular labels of similar images"). I think this could be quite useful, especially when we hit the hundreds of labels mark.

But not sure yet how we can best detect the scene type. Some possibilities:

• add possibility to specify scene labels. I guess we should handle scene labels slightly different; it probably doesn't make sense to create a annotation task for scene labels.

• try to detect the scene type from the labels. if the image has a significant number of urban scene labels, it's probably a urban scene..so we could show the user other urban scene labels.

• use image hashing to detect similar images.

[dobkeratops]

if we could detect the scene type(s) and give users some label suggestions

that makes sense .. could that be completely generalised by extending the 'most common labels' to 'most common labels co-occuring with the current labels..'

e.g. starting to add labels in a scene with nothing, you'd just have "the most common labels", but then as soon as you add 'road' it would switch to showing "the most common labels that co-occur with road.." and so on

without going as far as figuring that out from the data, that could indeed be done as preset tables of common groupings (all the urban labels; all the kitchen labels; etc..).

[bbernhard]

that makes sense .. could that be completely generalised by extending the 'most common labels' to 'most common labels co-occuring with the current labels..'

good idea!

without going as far as figuring that out from the data, that could indeed be done as preset tables of common groupings (all the urban labels; all the kitchen labels; etc..).

that's a good idea! At first, I thought that it might be even better to correlate the existing image labels with labels from other images. That way we wouldn't need to maintain the label groupings ourselves. But on a second thought, I think that's quite extensive and adds unecessary stress to the database.

Coming back to one of your first suggestions: scene labels

Do we still want to have scene labels or has the system evolved in a way where we don't need them anymore? I think at least for the label grouping, scene labels could be quite useful. I guess we could also use this mechanism to group images depending on the use case (e.q self driving car). Maybe we should use a more general name then? (metalabels?)

Given that we still want to have scene labels/metalabels (or whatever name we come up with), I would propose the following concept:

(in lack of a better name, I'll refer to the new label type as metalabels)

• metalabels should be displayed differently in the UI
• metalabels aren't annotatable
• metalabels can be validated in the validation view (similar to normal labels)
• metalabels should be queryable (i.e we can use them in the explore view)
• similar tending -> productive labels concept. (if we want to keep only one textfield for label input, we need to come up with a way to distuingish a normal label from a metalabel; maybe add the (meta) suffix?
• metalabels and normal labels are two disjoint sets. so a label x is either a metalabel or a normal label - but it can't be both. Does that always hold true? If not, we probably need to come up with a distinction between normal labels and metalabels in the dropdown list (either a different icon? or a (meta) suffix?)

Of course, all those points are open for discussion - that was just a quick braindump ;)

[dobkeratops]

Do we still want to have scene labels or has the system evolved in a way where we don't need them anymore metalabels should be queryable (i.e we can use them in the explore view)

i'd say not strictly needed; A 'scene label' would now just be a name for a cluster of labels? With the longer label list, I think you can specify enough now. "road+building" pretty much makes a street. you can cut-paste this with the comma seperation. it might be a nice shortcut to have, but of low importance.

The fact that we can submit batches of photos with the labels pre-assigned means there's a way to deal with this offline.. you can quickly get a backlog of images with enough key labels awaiting annotation

I suppose it could indeed be nice in the explorer ... imagine if 'street' made a shortcut "road&(building|pavement)" or something like that

[dobkeratops]

found this good example of 'parking space'.. there's a distinct material , it's certainly not pavement, but it's not clear if you'd call it road either..

[dobkeratops]

this is an example which could be handled with driveway or path .. here the road flows into a park.. but it's not drivable (bollards prevent entry)

[dobkeratops]

here's an example for carpark/driveway, .. the inner area (above the annotation) is connected to the road but doesn't lead anywhere and mixed use with pedestrians

[bbernhard]

A 'scene label' would now just be a name for a cluster of labels?

yeah, more or less. I guess we could also use this mechanism to tag images - in case we want to focus on certain use cases (see also this discussion here: #45). I guess we don't necessarily need scene labels here, as there is always the possibility to form explicit queries like: (road & building) | road | street light | speed bump | pavement ...). But I think it gets more and more complicated the more labels we have.

What I am missing a bit at the moment is the possibility to specify labels on a per-image basis that aren't annotatable. We already have that possibility via API (annotatable: false), but we don't have that yet exposed via UI. The only possibility at the moment is to add a label and then mark the corresponding task as un-annotatable or blacklist the annotation.

I think such a mechanism would open a lot of possibilities:

• Describe what you see with natural language (complete sentences?). I am wondering if a neural net can be trained to tell blind people what's shown on an image.
• describe the weather (sunny/cloudy/foggy...)
• describe the atmosphere of an image (calm/relaxing...) - mostly subjective, but I guess that's also okay

[bbernhard]

I suppose it could indeed be nice in the explorer ... imagine if 'street' made a shortcut "road&(building|pavement)" or something like that

that's an interesting idea. I guess the same principle could be also applied to the label graph.

here's an example for carpark/driveway, .. the inner area (above the annotation) is connected to the road but doesn't lead anywhere and mixed use with pedestrians

I am wondering if we can use the (quiz)refinement mode here. ("Is it driveable?")

But that would require that users can extract that information from the picture and I guess that can be quite hard, if you do not know the area (it's probably easier for you, as you've uploaded most of the urban scenes)

[dobkeratops]

I am wondering if we can use the (quiz)refinement mode here. ("Is it driveable?")

i guess the label graph could actually handle all this - 'man made area' ->{ purpose:{'drivable area' , 'walkable area' ..} materials:{'asphalt','paving stones','cobblestone',..} }-> {'road', 'pavement' , 'driveway}

require that users can extract that information from the picture and

right.. certainly not always possible, but human intuition can figure out some useful information

[dobkeratops]

just doing some 'river' examples , i'd suggest we can make some similar occlusion assumptions (i.e. surface types like road, river.. are occluded by other annotations); we've already got 'bridge', 'pillar' would help some, 'fence'/'railing' , 'fence post' .. and also perhaps "pier", "jetty" https://en.wikipedia.org/wiki/Jetty

there's also some examples of rocks.. tried to draw around most of them, but maybe you could mark rock in water'/'rock near water' or something like that (hints for spray textures, bits of reflection...)

[dobkeratops]

another surface label request is "puddle",.. there's some examples of blobs of water on road/pavement around rivers or parks .. nice to distinguish different water types and give scale hints.

[dobkeratops]

this is the classic example of 'driveway' between 'pavement' and 'road'. There's another label request here - gate

[dobkeratops]

this is a good example for path, it looks alot like you're on a road, however from context you can see it's taken inside a park, closed off with bollards to prevent entry by cars.. the real road is infront

[dobkeratops]

few more labels that could be useful: ditch https://en.wikipedia.org/wiki/Ditch road_verge https://en.wikipedia.org/wiki/Road_verge girder https://en.wikipedia.org/wiki/Girder pillar https://en.wikipedia.org/wiki/Column (pillar+wall, pillar in bridges, ..) tunnel https://en.wikipedia.org/wiki/Tunnel cave (in natural scenes) https://en.wikipedia.org/wiki/Cave post box https://en.wikipedia.org/wiki/Post_box litter bin, waste container https://en.wikipedia.org/wiki/Waste_container skip https://en.wikipedia.org/wiki/Skip_(container) traffic cone https://en.wikipedia.org/wiki/Traffic_cone barrier: ( i would start with the general word.. so many types..)

• boom barrier https://en.wikipedia.org/wiki/Boom_barrier .. common in carparks
• traffic barrier https://en.wikipedia.org/wiki/Traffic_barrier
• handrail https://en.wikipedia.org/wiki/Handrail
• temporary barriers common in construction sites

pallet wooden platform for carrying stuff https://en.wikipedia.org/wiki/Pallet

steps,stairs https://en.wikipedia.org/wiki/Stairs .. connecting different vertical levels, sometimes found infront of buildings escalator https://en.wikipedia.org/wiki/Escalator found in pedestrian areas and stock pictures of malls etc

statue https://en.wikipedia.org/wiki/Statue .. important to distinguish false positives for 'person' related monument, sculpture ..arty objects around cities

[dobkeratops]

good example for traffic island, there's a mixture of materials/surface types.. but only part is paved, or intended for pedestrians. 'traffic island' would cover more of the blob here between the road and paved part. dirt or soil might be another useful one - the parts there could be marked dirt/grass ?

[dobkeratops]

material type: dirt or soil, again being bendable with grass to handle the common case of 'patchy grass' https://en.wikipedia.org/wiki/Dirt https://en.wikipedia.org/wiki/Soil (soil might be the more accurate name, although i often hear of 'dirt road' etc https://en.wikipedia.org/wiki/Dirt_road .. imagine if you could just blend dirt/road

[dobkeratops]

here's an area with a mix of dirt/pavement or soil/pavement...

dust, gravel would be nice too https://en.wikipedia.org/wiki/Gravel ... this is very common in construction sites

and we've got some of this in urban features shale https://en.wikipedia.org/wiki/Shale

[dobkeratops]

this is possibly dirt/foilage ? or dirt/bush ? to the left maybe you'd call it bush/grass?

[dobkeratops]

here's an example which could have been handled with grass/dirt .. i'd tried to draw a few blobs to 'hint' patchiness, but it would be more efficient to say the area is a mix

[dobkeratops]

monument distinct from buildings in that they have no interior. (monument, statue, sculpture would handle these decorative/arty objects.. again label-blend would handle the crossovers. monument/statue, statue/sculpture, monument/building for ones which are ambiguious)

[dobkeratops]

example of "hard shoulder" found in the current image set .. I didn't get any of these myself, because I'm on a bicycle... these are only found on motorways, not bike friendly :) https://en.wikipedia.org/wiki/Shoulder_(road)