Suggestion - mirroring - left/right labels

[dobkeratops]

Distorting data to amplify the training set is a common trick,

li’ve done some annotations using left/right (ie toward pose estimation etc) Eg search for left/man right/man keft/woman right/woman(I can only see these through unified->browse +> rework, not explore, because of the unofficial use of / I guess)

https://imagemonkey.io/annotate?mode=browse&view=unified&query=left%2Fman%7Cright%2Fman&search_option=rework

Maybe there would be a way to officially specify that certain labels are not safe to mirror, and would be flipped if you did (I.e left swaps with right). Maybe the site could have an option to do this automatically on download, or the labels could be listed with a mirrorable flag

[bbernhard]

Many thanks for the suggestions!

Maybe there would be a way to officially specify that certain labels are not safe to mirror

Do you a particular workflow in mind here how that could be expressed in the UI?

[dobkeratops]

So this doesn’t affect you when annotating at all… unless you started adding ways to register labels (currently you manually approve them, and that’s ok)

It’s just something anyone using the data needs to know about.

I know you’ve already got something to automatically train - this could do mirroring and the appropriate left/right swaps

alternatively it could list this flag with the label list in any situation where you can export raw data.

I can think of 3 settings: mirrorable/default , “swaps when mirrored” (ie left,right), and “not mirrorable” (eg annotation of text, might be important for signs? Also for driving.. cars in specific territories drive on a specific side of the road)

if that’s overkill… just mention this as a comment in the documentation (ie if it is just left,right )

[dobkeratops]

There’s now 1000 annotated examples of left/right pairs (500 man,500 woman)

It might be nice if these could be added to the official label list, although I think these are harder do do than “left/arm” “left/leg” etc. I’m counting on the ability to just overlap different information

(Individual left/right limbs hands etc end up making the label list much more complex)

[bbernhard]

It might be nice if these could be added to the official label list

That shouldn't be that much work, I'll have a look at this on the weekend :+1:

alternatively it could list this flag with the label list in any situation where you can export raw data.

Ah, I see. Yeah, that definitely makes sense. I am wondering, if we can "hardcode" that. So, if a label starts with left/ or right/ it is automatically non-mirrorable or if it is better to manually curate that list?

Eg search for left/man right/man keft/woman right/woman(I can only see these through unified->browse +> rework, not explore, because of the unofficial use of / I guess)

oh, there seems to a bug in the "explore" view. I'll have a look at that - thanks for the info!

[bbernhard]

short update: the bug in the "Export View" should be fixed, but before I can push the changes to production it probably will still take a bit. First I want to update the PostgreSQL version on the server (which is long overdue). As this is quite a bit of work and requires some downtime, I'll probably do that in my vacation. That way it is easier for me to fix stuff in case the database upgrade results in some errors.

[dobkeratops]

There’s now 2000+ left + right pairs (roughly 1000 each of man/ woman)

I think you had an automatic way to start training for a particular label set? This would be interesting to try .. although usually they use much larger sets for tasks like this.. Left/man , right/man, left/woman, right/woman

doimg a multi- task net (some alternate branches of the same core net) that also used the arm, leg, head, face, torso, hand, foot annotations would also help it, but that’s more complex to arrange. finally, between all the different ways in which man,woman are annotated , (labels like “man/sitting, walking,running ) , there’s 8000+ human outlines in this dataset now

[bbernhard]

I think you had an automatic way to start training for a particular label set?

I was renting a GPU instance for about a year, but unfortunately they increased prices and it got more and more expensive. At some point I couldn't justify the expenses anymore, so I had to let go. But I've heard that people are selling their GPUs as mining isn't that profitable anymore. So maybe I get a used card at some point (I think that should be way cheaper than renting a bare metal instance)

[dobkeratops]

Ok I’ve got a 3080 here I see the guide on how to run it (I’ll need to figure out docker.. I haven’t used these web tools). I’m guessing if I look for the script that downloads and trains that will show how to grab the images & annotations raw aswell. Maybe a nice addition would be a “developers export” in the UI which gives you a zip file, not sure if that’s practical.

I was also curious to try and write something to browse the dataset , draw thumbnails of the annotations

I’ve got some basic web hosting myself but don’t have a domain name. I’ve been messing with wash+rust +webgl lately . This could have some interesting possibilities for a fancy viewer (remember the ideas like an image galaxy, or we could just try to do and overview where you can see the annotations,etc)

If you’re curious - my codebase runs on desktop & web, I’m making a game engine in Rust now https://vimeo.com/677060352 this uses my own improvised low poly art, I’m hoping some day a NN can generate texture and detail. Hence the interest in a dataset with environment peices, human figures etc

[dobkeratops]

there's now just over 3000 total left/right side of the body annotations (~1500 x man x (left|right), ~1500 x woman x (left|right)). a smaller subset of those have hands/arms etc, probably more than half have the head/face