Project Roadmap - What's next?

[bbernhard]

In the past few weeks, I was mainly working on infrastructural topics. In particular:

• docker image improvements
• more integration tests
• small bug fixes
• travis ci integration
• a few small changes regarding component parts (will be pushed to production soon)

As I am now almost done with most of the stuff, I have again more time to work on new features/bugfixes. Is there something we should add/change, that would result in a huge improvement?

At the moment, we have quite a lot of tickets in the backlog - all of them containing some really cool suggestions + ideas. What I am struggling a bit with is to prioritize all those tickets - if I could, I would love to tackle all those ideas at once. So I thought about creating this ticket here, in order to talk a bit about the next steps and the direction we wanna head.

Any suggestions are really welcome!

@dobkeratops adding you, as you're using the service frequently :)

[dobkeratops]

nice, I know how a project needs cleanup phases.. clear out any 'technical debt' . The question is what additions will inspire more users to join ?

I think the core functionality is pretty good now, the range of fixes earlier (plus getting a longer label list) has really helped.

I would guess visual ways of exploring whats already there might help inspire people to join (eg the browse modes- eg imagine if the browse mode started with an image per label.. visually appealing whilst still allowing fine control through the operator queries..)

the other avenue is to find people who have use specific labelled data cases already.. my interest in this has really been very broad and open ended, driven by a general belief that we need these kinds of open platforms for data-driven collaboration (to avoid a few monopolies controlling everything) - so i've been happy to support the project by giving it regular use in spare moments.

We've talked about things like calorie calculators and Self-Driving cars speculatively but I'm not personally working on either (incidentally did you see the news story about a chinese implementation of the former? I think it was a fairly limited buffet-checkout example taking photos of plates from above.. worth researching..)

[bbernhard]

The question is what additions will inspire more users to join ?

good question. I've to admit, that I was a bit lazy the past few months and haven't actively promoted the service on reddit & co. I guess that could probably help to get more users on board.

Something that's also on my todo list are stresstests. As there are currently only a handful of users active at any time, it's really hard to say how the system behaves with much more users. In theory, it shouldn't be that bad, as e.q images are cached and served by cloudflare. But I am sure there will be something that will break...:D

I think the core functionality is pretty good now, the range of fixes earlier (plus getting a longer label list) has really helped.

that's great to hear!

I would guess visual ways of exploring whats already there might help inspire people to join (eg the browse modes- eg imagine if the browse mode started with an image per label.. visually appealing whilst still allowing fine control through the operator queries..)

that's a nice one, thanks! I'll have a look at it :)

the other avenue is to find people who have use specific labelled data cases already.. my interest in this has really been very broad and open ended, driven by a general belief that we need these kinds of open platforms for data-driven collaboration (to avoid a few monopolies controlling everything) - so i've been happy to support the project by giving it regular use in spare moments.

We've talked about things like calorie calculators and Self-Driving cars speculatively but I'm not personally working on either (incidentally did you see the news story about a chinese implementation of the former? I think it was a fairly limited buffet-checkout example taking photos of plates from above.. worth researching..)

yeah, the open source aspect and being indepent from the big monopolies is also was motivates me the most.

You are right, having a few concrete use cases would definitely be helpful - still looking for a use case which is broad enough so that a lot of users can contribute to and where we we can easily gather large amounts of images.

[bbernhard]

it's been a while since the last update, so here's a short one:

• the changes that were needed to fully support component labels are now done. i.e i'll make the first component labels (head/person, head/dog,..) productive in a few days.

For now, I went with the dot-syntax (i.e: dog.has='head' instead of 'head/dog). I totally get, that the slash notation is easier to type when adding a new label, that's why the slash syntax is still supported. When a new label with the slash syntax will be added, it will automatically be converted to the dot syntax.

• As already mentioned before, I would like to release pre-trained models on a regular basis for image classification, object detection and object segementation that were trained on the ImageMonkey dataset. In order to keep it simple, I started with an image classificaition model based on inception v3 and trained on the follwoing labels apple, person, tree, car, building, cat, dog, pizza via transfer learning. To avoid training bias, I limited the number of images per label to 300. (we have much more people and tree images than pizza images and I was afraid that this would bias the neural net)

Here's the site, where the models will end up: https://imagemonkey.io/models

What's still not clear to me: How can we measure the model's quality/progress? Let's assume, we automatically create a new model every month. How can we make sure, that our newly trained model is better than the one from last month? Is there like a "standardized" image test suite, that we can run our model against?

I guess we could create one ourselves, but I think in order to create a good test suite, we probably need to consider also quite a lot of the recent fooling techniques (e.q: one pixel attacks)

[dobkeratops]

awesome.. re. testing a model, i'm not sure what the formal methods used are .. i would guess something like

• pick a random 75% of the dataset for training, train on it, and use the other 25% to test it?

r.e. "dog.has='head'" etc .. right - having something unambiguous as the underlying representation is better; allowing it to be reached by a shortcut is the best of both worlds.

[bbernhard]

pick a random 75% of the dataset for training, train on it, and use the other 25% to test it?

yeah, something similar like that I am doing at the moment. Currently, 75% of the dataset will be used for training and 25% for validation; both the images for training and validation will be randomly picked.

However, I am a bit worried, that those randomly picked validation images make it really hard to compare the resulting models. e.q: We have quite a lot of "high-res" dog portrait photos. I could imagine, that a neural net will be able to classify those type type of images pretty fast, without much training data. On the other hand, there's also a significant portion of dog photos, where it's harder to spot the dog (because of bad photo quality, the dog's position,...).

I guess the model's accuracy then heavily depends on the type of images we randomly select as validation set. What I was hoping for is, to find a "standardized" image test suite or a way to create one (e.q: add noise to images, flip images, change random pixels in images, ...).

[bbernhard]

The idea would be to make it really hard for the neural net to classify the image correctly and to create as much false - positives as possible. Once we have that in place, we have some sort of metric that (hopefully) helps us to make the model perform better. e.q: I would really be interested to see which actions result in the best improvement of the model.

Is it the sheer amount of training data that makes the model perform better? (more training data -> better accuracy?) Or does it help if we add noise purposely to training images? Does it help, if we have different type of images (normal ones, fish eye...)? Can we increase the models accuracy, if we feed the model only the relevant image crops or does it then get worse?

[bbernhard]

another short update:

• the component labels integration is now done (already made the first component label, head/person, productive)

• added a new type of label called metalabel. Metalabels are per default not annotatable (if it turns out that this is too restrictive, we can weaken that later). The metalabel implementation is completely transparent...i.e the labeling works the same as before. The only difference is, that metalabels are marked as non-annotatable (and therefore won't show up for annotation). e.q: kitchen is a metalabel

• over the christmas holidays I was experimenting a bit more with gamification. It basically ended with a small android/iOs app. The app basically works like this:

• you sign up with your ImageMonkey credentials

• you get a list of all productive labels

• if you click on a label, the smartphone camera opens and and you can snap a photo that gets then uploaded

• for every uploaded photo you earn a star or optional some special badges

• Game Rules

[dobkeratops]

hi, (been away for Christmas, back now) gamification - interesting idea, should be great for casual phone users (i.e. most people)

[bbernhard]

hi, (been away for Christmas, back now)

welcome back! hope you had a nice vacation :)

gamification - interesting idea, should be great for casual phone users (i.e. most people)

yeah, I hope so too.

Initially I wanted to do something similar to the Google Image Labeler project (https://en.wikipedia.org/wiki/Google_Image_Labeler https://www.youtube.com/watch?v=yWZUxnRYISs), but I wasn't sure if we can attract enough players for those 1:1 battles.

I am still looking for a good gamification idea that makes the annotation process a bit more interesting. In my opinion labeling, validating or donating a photo is way more interesting than annotating stuff. I am not sure why I am feeling that way, but I guess it could be because those tasks require a shorter attention span (you get faster results).

If we could gamify the annotation process I think that would be really awesome.

[dobkeratops]

I guess it could be because those tasks require a shorter attention span (you get faster results).

exactly that. it's easier to do them casually. annotating requires more concentration. But more data is always better; you could still train on the image wide labels.. if you can get more simpler contributions, great