Suggestion.. simplify main UI once unified mode has add labels..

[dobkeratops]

So at the moment the main UI has got quite complicated, I think this is a consequence of the evolutionary path .. maybe you could reduce this to a “unified/search mode” for users who want to guide their work, and an “automatic/task mode” for casual users who want the system to guide them.(not sure what the best names are)

.. ie simplify the entry point to the modes based on this one choice..

If a user has a specific task or domain in mind (“road vs pavement for self driving cars”) they can use the unified mode and do the annotations that interest them - even if the label list is long, it’s just another part of the search process (if they had searched images they would still go through some sort of filtering process).

Conversely maybe you can also streamline the “task mode” by alternately serving ‘add labels’ and ‘annotate’, but you could bias the rate depending on the current state of the database (if there are fresh labels with no annotations, serve those.. if there are a bunch of new images without any classification, serve those maybe hinting to ask for scene labels)

Would an approach like this allow you to simplify the individual screens out a bit aswell - eg does the unified view remove the need for the “go to” shortcuts?

I’ve omitted “describe image” in this, i kind of hope this could be put in the unified view aswell eventually there could be an editable line of text above the image.. or it could be compacted in the label list (or scene label list?) maybe it would get too crowded though

I had tried to make some labels which combined description with labels using square brackets to divide things up, eg [[man] riding [elephant]], [[dog] in [garden]] ... (nesting.. you could annotate the man plus elephant , and man and elephant as individual objects.. ‘dog in garden’ is a typical main label that describes a whole image, yielding ‘dog’ as a labelable component) but I’m night sure how those would fit in the label menu. But Those are actually easier to write than a whole image description (trying to string together a whole sentence .. if it’s a very complex scene you can break it into pieces)

[bbernhard]

Love that!

thanks a lot also for the mockup - that looks really cool! ;)

If no bugs occur during the final testing tomorrow, then I'll push the first version of the "unified mode + add labels" to production. That means I could start right after that with the simplification :)

Just a short question (more questions will for sure follow when I am working on that): should we let the "default (simple) mode" die (at least for non-mobile devices) and make the "browse (expert) mode" the only mode?

With your new proposed menu structure we would have this two new top level entries:

• Unified Mode
• Task Mode

Which submenu entries would the Task Mode have? Should we add every possible combination? i.e:

• Annotate (Default)
• Annotate (Browse)
• Label (Default)
• Label (Browse)
• Validate (Default)
• Validate Browse)

Or does it make sense to get rid of the default mode and make the browse mode the only mode? In that case we could simplify the menu to:

• Annotate
• Label
• Validate

I guess we could make the browse mode more user friendly by adding a "I am feeling lucky" button which displays random tasks.

Could we get rid off the default mode that way?

Although I have to admit that the default mode is quite nice for labeling...for labeling I really prefer the default mode over the browse mode.

Or is there maybe another way how we can simplify the number of menu entries while keeping both the default and the browse mode?

[dobkeratops]

I guess we could make the browse mode more user friendly by adding a "I am feeling lucky" button which displays random tasks.

I wanted to suggest going further than that:- start it with the search box saying “random”, so the first thing you always see is a page of random images ready to click straight into work.. and you just replace the word random with a specific term if you didn’t find an interesting image from the random set. You could have a few special search keywords.. “random, recent, unlabelled, ...” but true random is probably the best default

1 load the site, 2 click “unified search mode..” 3 see random images.. click an interesting image.. 4 in unified view: enter or click a label, if the one it started with isn’t interesting. 5 in unified view: annotate 6 go to 4, or get back to the search view..

Right.. unified(search)mode and (automatic) task mode . At step 2 some users will go a bit further and enter a specific search.

Which submenu entries would the Task Mode have? Should we add every possible combination? i.e:... Or does it make sense to get rid of the default mode and make the browse mode the only mode? In that case we could simplify the menu to:

Right.. my reasoning is if you want to make such specific choices, you’re better served with the unified mode; Task/automatic mode is so you can just sit back and let the system guide you.

... So I would keep the task mode menu as simple as possible.. the best being no submenu.. you go straight into a stream of random tasks, whatever the system deems is needed.

that way you can satisfy both casual and expert users. You could swap the order too, show task mode first.

Could validation happen as part of the autoamtic system guided stream? I’m sort of hoping validation can eventually be made more interesting by combining with refinement(“what type of car is this? Saloon,suv,not a car..” ... every refinement validates the previous information. Keep 2 records, ‘the recent given information,and how far validated it is..’), or allowing people to do corrections (downvotes?) as they encounter things in unified view.

[dobkeratops]

Ok I just tried the latest version .. I see the add label box in unified mode.. great to see, the goal is in sight.. Using it on the iPad, I found it didn’t seem to always work , although it did once. I will just try on the desktop again). There’s also a residual problem where if you delete anything it sometimes won’t let you annotate something else after.

Best to keep the whole menu as you have it until this is rock solid IMO.. once this is solid then it will allow the simplification. It might need the seperator/add hot key aswell.. (it will certainly be pleasant to add a bunch of labels with hands on the keyboard then flick between them with hands on the mouse etc to draw)

[bbernhard]

Using it on the iPad, I found it didn’t seem to always work , although it did once. (I will just try on the desktop again).

that's strange :/ I haven't tried it on a tablet yet, so maybe there is something broken in the tablet version (will check that next). Would be great if you could give it a try on your desktop machine to check whether it's really just the tablet version that's buggy :)

There’s also a residual problem where if you delete anything it sometimes won’t let you annotate something else after.

I'll have a look tomorrow to see whether I can reproduce that in my dev environment. If you recognize a pattern that helps tracking down the bug, please let me know :)

Best to keep the whole menu as you have it until this is rock solid IMO.. once this is solid then it will allow the simplification. It might need the seperator/add hot key aswell.. (it will certainly be pleasant to add a bunch of labels with hands on the keyboard then flick between them with hands on the mouse etc to draw)

totally agreed. I think we might also need a bunch of hotkeys that make the navigation between the different UI elements a bit easier. (switching between labeling/annotation feels bit sluggish at the moment)

[dobkeratops]

Right.. just tried it on the desktop .. works fine! This is a HUGE step forward! This is the best it’s been, trying to think what might be different on an iPad browser.. there are little differences in how it responds to focus (eg it scrolls the list differently), ...but I would worry so much. The fact this works on a desktop machine or laptop is great. (after breaking my laptop ive been using my desktop a lot more)

we still need the dedicated add labels screen for label suggestions.

I think we might also need a bunch of hotkeys that make the navigation between the different UI elements a bit easier. (switching between labeling/annotation feels bit sluggish at the moment)

Sure. It’s good enough to be worth using right now, but yes if you had similar “add label” hotkeys and seperator detection to the add label screen , and label toggle hotkeys (to work “dpaint style”.. like the palette switch hotkeys) that would be perfect. You could put you hands on the keyboard, add a bunch of labels, then put one hand of the mouse and one hand on the label toggle, and annotate really quickly

[bbernhard]

Right.. just tried it on the desktop .. works fine! This is a HUGE step forward! This is the best it’s been, trying to think what might be different on an iPad browser.. there are little differences in how it responds to focus (eg it scrolls the list differently), ...but I would worry so much. The fact this works on a desktop machine or laptop is great. (after breaking my laptop ive been using my desktop a lot more)

Thanks, that's great to hear!

I've just checked with my Pixel C tablet and it seems that there big differences between the various browsers. With firefox (my default browser) the site is almost unusable on the tablet. Drawing a rectangle is really sluggish and slow. There also seems to be a rendering bug when adding a new label in the unified mode (sometimes the newly added label won't show up although it is there; a touch on the labels list usually triggers a re-rendering of that part which fixes the problem).

With the chrome browser however, everything works fine.

Really strange...

[dobkeratops]

Bearing that in mind, perhaps in any UI simplification you do want to keep the various modes available as woerkarounds.. I don’t know what’s easier to maintain, more simpler screens or one sdvanced screen. I could still imagine thededicated “add labels” screen being useful with its large area devoted to.popular label shortcuts (imagine that furthe guided by scene label correlation.. pick a scene or category first, then get label suggestions for that..)

And there is of course phones to consider.. I haven’t tried using it on a smartphone, which is what most people on this planet probably use most of the time now. (I imagine the original task idea is probably a solution to fitting enough work on a smaller screen.. no room for a label palette plus workspace? .. so you probably do want to keep it in parallel..)

Anyway.. lots of options to explore.