add 'donations' to activity chart?

[dobkeratops]

donations could potentially contribute training data directly (e.g. if you can assume the given label is the most salient description of the image - I'll mention elsewhere what I've done locally to classify the uploads I did in directories)

[bbernhard]

Could you please elaborate on that more. I am not sure if I completely get it ;)

[dobkeratops]

basically once you have the free-labelling option, it will be possible to give a useful training label for the whole image .. at that point , the act of uploading an image (with richer or more accurate description) would itself be enhancing the usefulness of the dataset. What I'm doing locally after upload is keeping the images classified in directories under names like 'park', 'street', 'alleyway' etc - I want to try using those as a whole-image label; perhaps whilst running out of different styles of photo to take around here I also started taking 'texture' style images(e.g. plain picture of walls/fences etc) which would be classified at the image level (nothing inside to really annotate) e.g. "new brick wall","stone wall" ,"concrete wall" etc . The next question is will we be able to retro-actively submit those extra labels (I notice the site tells you if you uploaded the same image twice; thats really useful - perhaps if you try to upload again but with a more accurate label, it can add that.. but I can make my sorted images directly available aswell)

[bbernhard]

got it! That could indeed be pretty useful - which brings me back to the whole metalabel/scene idea of yours

Would it be a valid workflow to extend the free labeling concept to also include the possibility to specify whether you want the label to be annotatable? I guess such a metalabel concept could be pretty useful, as you could inject a lot of useful labels (weather, country, texture information). As the free labels aren't productive by default, we could continously go over the free labels list and make them productive together with the possibility to query those nicely. (e.q weather = 'sunny' | weather = 'cloudy')

If we go with a more structured format here (e.q JSON) for the "per-directory labels file" , then I could imagine that we could easily add that property there as well.

(I notice the site tells you if you uploaded the same image twice; thats really useful

yeah, it uses image hashing to check whether an image is already uploaded or not. That works pretty well if you upload the same, identical image twice; it should even work if you rescale the image.

perhaps if you try to upload again but with a more accurate label, it can add that.. but I can make my sorted images directly available aswell)

What we could probably do, is to return the uuid of the image that's already in the database as part of the HTTP response. Given the uuid you could then later use the free labeling API endpoint to add more labels to that image. I guess that could work. So if you aren't running out of disk space, it might be worth to keep those images structured as they are ;)

[dobkeratops]

to also include the possibility to specify whether you want the label to be annotatable?

that would be useful, yes; perhaps later extend with 'classifiable' (it can be further refined via quiz. "wall" -> "new brick wall" etc). Another division in labels is "countable" (e.g. car, person)/"non-countable" (road,pavement,grass,water- surface types for area coverage). (and we already covered how the plural form of a countable-label might turn into non-countable

I guess such a metalabel concept could be pretty useful, as you could inject a lot of useful labels (weather, country, texture information)

definitely. i was going to mention the camera lens (fish eye or normal), i'm sure many users will want to know that (e.g. if training a net which is going to be used with one or the other.. plus I know fish-eye lenses aren't so useful for textures because they distort so much. ) 'country,weather' again might be great training signals (force a net to pick up on subtle details)

So if you aren't running out of disk space, it might be worth to keep those images structured as they are ;)

correct - disk space isn't a problem. I can buy hard-drives faster than i can find things worth photographing.

[bbernhard]

that would be useful, yes; perhaps later extend with 'classifiable' (it can be further refined via quiz. "wall" -> "new brick wall" etc).

sounds good.

Another division in labels is "countable" (e.g. car, person)/"non-countable" (road,pavement,grass,water- surface types for area coverage). (and we already covered how the plural form of a countable-label might turn into non-countable

yeah, right...that's also on my todo list.

definitely. i was going to mention the camera lens (fish eye or normal), i'm sure many users will want to know that (e.g. if training a net which is going to be used with one or the other.. plus I know fish-eye lenses aren't so useful for textures because they distort so much. ) 'country,weather' again might be great training signals (force a net to pick up on subtle details)

I think I am now almost done with the API token support integration in the backend. So I'll probably start with the extension of this script: https://github.com/bbernhard/imagemonkey-libs/blob/master/python/snippets/donate.py

As you probably have used the script more extensively than I am, a few short questions:

• is the folder approach flexible enough for your use case? If you don't have any objections, I would keep the general folder approach in place, but just extend it with the possibility to specify multiple (meta-)/labels per folder.

• given, that we keep the folder approach: Do you have any preferences regarding the format of the label file that will be placed inside every folder? I would see the following possibilities: Either we use a single JSON file which contains a list of all labels including the information whether the label is of type metalabel (or whatever we name it) or not. Or we could also use two different text files: e.q label.txt and metalabels,txt - the labels will then be separated by a newline. I guess the text files approach could be better, if you create those files per hand? But I'll leave that one up to you - for me everything is fine.

[dobkeratops]

As you probably have used the script more extensively than I am, a few short questions:

actually I have an admission , I couldn't get the script working so I was copying them in - at the point of each image I selected the best approximate imagemonkey label, then (sometimes) copied it to a more accurately labelled directory

is the folder approach flexible enough for your use case?

• yes. It's the OS's & file-managers job to make moving things between folders pleasant, and it seems they do ok

• but just extend it with the possibility to specify multiple (meta-)/labels per folder.

that would indeed help further

Either we use a single JSON file which contains a list of all labels

I'm happy with either, so pick whichever is easiest to code. What I might say is JSON sounds futureproof. (one meta-info file and then you can edit whats there) If the script can verify the format and remind you ("it doesn't look like .. - the correct format is ...") - that will work fine

[dobkeratops]

( actually come to think of it, r.e. the "classify label" option, perhaps that should be default for all labels (no need to specify per upload), if we could get a permanent label tree or label graph going)

[dobkeratops]

this is what my current directory structure looks like.. it's a compromise with overlap - cases like 'street' and 'cityscape' are catch all that carry the same elements, just a different emphasis .. when one item dominates it go in 'buildings', 'vehicles', 'road' etc

'vegetation' splits into bush tree grass .. there's probably more to come there e.g. hedge. I figure these are important distinctions for a vision system , giving scale (trees and bushes are very similar at the pixel level but are usually more easily distinguished in context)

Adding any of these as labels would help me out with future uploads.. the weather is great at the moment and I can get 1000+ images in a day

walls might seem mundane but what I have in mind is a neural net for figuring out repeatable texture-application for scans of urban scenes, so I see splitting that up into new/old brick, stone, concrete ; again a 'wall' is often a building element (and often indistinguishable when zoomed in) but the absence of doors/windows separates them. There will also be 'surfaces' for the same purpose but at the moment there's just 'pavement/road/grass' elsewhere (as I mentioned I think road vs pavement is really important for SDCs).

construction covers both building sites, buildings under construction, and construction machines (i.e. often found together) but construction machines would be an important subdirectory there (and really I find this is stretching the limits of a tree structure.. really there should be 'machine -> {vehicle , construction machine} etc' but this is an easier way to group the photos.

park contains trees, paths.. absence of road/pavement, vehicles.

water needs to split into river, sea, lake, canal, but I haven't sorted those yet. There would also be puddle as an extremely important label (again contextual recognition..)

a few examples gate

bins

bridge really includes 'flyover'/'pedestrian walkway'

statue or monument or sculpture is really important, i.e. distinguishing these stone/metal objects from person, and covers some of the 'arty'/decorative objects you find in cities (there should probably be fountain in there too)

I might tweak 'alleyway / road / street' (one directory 'urban_street' with subdirs 'city_street','residential_street','alleyway' etc)

I haven't even started on road signs, traffic lights, street lights

[bbernhard]

Thanks for sharing your folders structure - that was a really interesting insight!

short update: I think I am now almost done with the whole labeling/API token stuff. I'll be doing some regression testing now (I really need to implement some unit/integration tests) and write some documentation for the new Python script. I think I should be ready in a few days (maybe even tomorrow)...so you could be able to give it a try then.

[bbernhard]

Ok, the production instance should now be capable of accepting arbitrary labels (in case an API token is provided). If you want to give it a try, here is a short introduction: https://github.com/bbernhard/imagemonkey-libs/blob/master/python/snippets/README.md

I would start with a small folder of just a handful of images to see if everything works and the server is accepting the token; If the token for some reason isn't valid (e.q there is a typo) the script would just upload the images without labeling (i.e the images would show up as unlabeled). The reason for that is, that the whole procedure consists of two steps: First, the image is uploaded and then the uuid from the servers response is used to label the image. The donation API endpoint is public and works no matter if a API token is provided or not. But the label API endpoint needs an API token, in case you are pushing a label that's not already a label known to the system (i.e a label that's already in the labels.json)

[dobkeratops]

awesome! I'll have a read and see how it works.

[dobkeratops]

ok awesome! I think I got it to work, but I had to doctor the python script a little here (probably 2.7 issues)

def _load_img_and_resize(path):
       #didn't seem to work with int max_dimension / int width or height
       #that did integer division returning 'scale=0'
    max_dimension = 1024.0 #px ..
..
       #int() needed to really convert it into an int
    new_width = int(round(width * scale_factor))
    new_height = int(round(height * scale_factor))
..
    print("resizing %f,%f by %f to %d, %d (max_dim=%d)" %(width,height,scale_factor, new_width, new_height,max_dimension))

I disabled the 'confirm', strangely it was telling me 'yes not defined'.. not sure why

def _confirm(files, labels):
#   line = input("Are you sure? [yes/no]\n")
#   if line == 'yes':
#       return True
#   return False
    return True

it's great to have this.. I'm wondering if it's ok to upload a batch which has mostly the same labels and rely on validation to eliminate the missing ones from individual files - the permutations would explode if making a seperate folder for each combination;

(e.g. so I tried out a few images from construction areas, with 'barrier'/'traffic cone' but there's one image without a traffic cone)

[
    {
        "label": "barrier",
        "annotatable": true
    },
    {
        "label": "traffic cone",
        "annotatable": true
    },
    {
        "label": "building",
        "annotatable": true
    },
    {
        "label": "road",
        "annotatable": true
    },
    {
        "label": "pavement",
        "annotatable": true
    }

]

.. so I've tried putting 'road'/'pavement' in; I see you have the 'annotatable' flag which is nice. I was in two minds about specifying 'urban' or 'street' or something like that as un-anotateable, but the information that it has 'road' and 'pavement' is un-ambiguous.

before I upload more , it's worth thinking about that a bit more.. maybe it would be worth splitting labels that are 'definitely there' from labels that 'might be there' ? ...or is the implication that they're all optional (awaiting validation?) - that's probably the best default.

now I realise there's another question on naming .. US (sidewalk) vs UK (pavement) ; I suppose we can figure out a system for synonyms later, or maybe I should have put 'sidewalk/pavement' which would make it clear to both audiences

[bbernhard]

awesome!

regarding the confirm: I'll have a look with my Python 2.7 installation. I really like the confirmation step - it saved my ass more than once (I tend to select the wrong folder; add typos to label names ;))

I'm wondering if it's ok to upload a batch which has mostly the same labels and rely on validation to eliminate the missing ones from individual files - the permutations would explode if making a seperate folder for each combination;

yeah, I guess that should be fine - the validation phase should take care of that :)

btw: if a new label will be used >= 20 times we should see a new issue here: https://github.com/bbernhard/imagemonkey-trending-labels/issues (if everything works)

[dobkeratops]

ok great (And I agree the confirmation step is worth having.. I'll look around a bit more. I'm a bit hazy on some python details)

[dobkeratops]

really weird.. python console -

>>> ret=input("enter something>"); print(ret)
enter something>yes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'yes' is not defined
>>> 

[bbernhard]

Looks like that Python 2.7 needs raw_input instead of input (see https://stackoverflow.com/questions/4960208/python-2-7-getting-user-input-and-manipulating-as-string-without-quotations)

[dobkeratops]

right i just found that.. is that something they deprecated in 3.0 ?

from stack overflow.. i'll try this..

This works in Python 3.x and 2.x:

# Fix Python 2.x.
try: input = raw_input
except NameError: pass
print("Hi " + input("Say something: "))

[bbernhard]

I think there are quite a few things that need to be changed for Python 2.7. You already mentioned the int casting and I think there is another thing regarding the unicode/str handling, which seems to be different in Python 3.

At the moment I am a bit torn back and forth, whether it makes sense to port the script to Python 2.7...as Python 2.7 support is coming to an end (in 2020, if I remember correctly).

[dobkeratops]

ok , i'll see if i can update, but I'm also curious to try the same script on both 2.7/3.0 installs myself to see if those workarounds handle it; the int/float stuff seems like it would be ok.. the string stuff might be a bit more subtle but perhaps 'try/except' can make a really_input=(...)

[bbernhard]

right i just found that.. is that something they deprecated in 3.0 ?

just tried via Python 3 console...looks like raw_input is completely gone in Python 3 (I get a NameError).

[bbernhard]

ok , i'll see if i can update

One thing I really like about Python is that Python 2.7 and Python 3 works really well in parallel. I have both on my system, as almost all machine learning frameworks require Python 3.x and never experienced any problems.

[dobkeratops]

the other thing I might try and look at.. I was already using a bash script to convert /png's to jpgs for upload (I think the site might have an upload rate limit , png's can stall it) - I might be able to roll that into the upload script.

I suppose I can also try adapting the script to use a directory list

"I have both on my system, as almost all machine learning frameworks require Python 3.x and never experienced any problems."

i'll definitely jump over to my linux desktop for all that.. now I've got all these photos and videos lying around I should give all that a go. Various ideas to try with video motion vectors etc etc..

[dobkeratops]

somethign else I was going to mention as a suggestion, perhaps in the 'annotate ..' page the label to annotate could be changed to a drop-box holding the given label-list (before I'd have suggested 'all labels') - it might smooth out the process of going through a potential set and having to verify 'there's no X..' ; but in my usage i'll try to stick to a few important high-probability items .. there's a load of images I can do next time which will have a few clear labels

[dobkeratops]

What about retro-active label application through this script - quite a lot of images have already been uploaded (just with car/person/building) - I note the site does know when an image does exist; would it be possible to make the act of trying to upload again apply the label retroactively?

(I suppose this is something we could go back to , e.g. if you get more examples you can curate the default inbuilt label list, and eventually they can be applied through "add labels.")

[bbernhard]

the other thing I might try and look at.. I was already using a bash script to convert /png's to jpgs for upload (I think the site might have an upload rate limit , png's can stall it) - I might be able to roll that into the upload script.

awesome!

I suppose I can also try adapting the script to use a directory list

yeah definitely. If you have a working solution, feel free to create a pull request. I think that could also be interesting for others. Maybe it's possible to enable that mode with an additional paramter in the script?

What about retro-active label application through this script - quite a lot of images have already been uploaded (just with car/person/building) - I note the site does know when an image does exist; would it be possible to make the act of trying to upload again apply the label retroactively?

I think it should be almost possible. The only problem is, that the uuid isn't returned in case the image already exists. But I can add that and push it to production during the next maintenance window (probably next weekend).

somethign else I was going to mention as a suggestion, perhaps in the 'annotate ..' page the label to annotate could be changed to a drop-box holding the given label-list

nice one! I think that should be fairly easily to do. I'll create a ticket, so that we can discuss it further :)

[dobkeratops]

so the site would send a response 'image already exists - uuid' - and at that point the script could use another endpoint to 'amend labels'? - or would you do it directly in the server

I can see that having the general ability to amend labels opens the door to alternate tools (I could revive my own experiments and make them work directly with your site)

[bbernhard]

so the site would send a response 'image already exists - uuid' - and at that point the script could use another endpoint to 'amend labels'? - or would you do it directly in the server

yeah, right. However, I would only see that as a workaround in order to make it possible for you that you can use the existing images to label them further. In general I think it would be more bulletproof if one stores the uuid that gets returned when pushing the image, to create the uuid <-> image filename association locally.

I can see that having the general ability to amend labels opens the door to alternate tools (I could revive my own experiments and make them work directly with your site)

that would be awesome - alternate tools are very much appreciated! I think that's one of the "unique selling points" we have against other, bigger datasets (like labelme). We can't compete with it's size (at least not now), but we can provide better tooling and libraries.

[bbernhard]

btw: it looks like that the github trigger works: https://github.com/bbernhard/imagemonkey-trending-labels/issues

If a label will be used more than 20 times, a github ticket will be created. I want to extend that feature, so that we can also see the label count in the ticket. I think that information could be useful when making the label "productive". A label that is used more than 500 times is probably more important than a label that's used only 20 times. - so we should tackle the former first.

My intention is to go over those tickets every week/two weeks and make the most popular ones productive (i.e add the labels to the labels.json). As soon as they are productive, everybody (i.e also users who aren't authenticated) can use them. When making them productive we should also talk about how we want to make them queryable. I guess for some labels we could implement different ways to query them: e.q given the label grass: we could use the label grass or surface = 'grass' or surface.type = 'grass' or all of the above.