Integrate deep learning object detection model for automatically labling

[vuthede]

I intergrate state-of-the-art deep learning object detection with Tracker to help us automatically labeling data. I create a new file for that named main_auto.py We can test this usability.

[Cartucho]

Cool! Let's merge the other PR first

[Cartucho]

@vuthede why did you close this one? It wasn't working?

[vuthede]

@Cartucho I realize that there is some minor bugs on that. It just works for people class, I have not made it flexible to label more than one class. Is it ok for me to re-open it with aforementioned issues remains or I should make it more perfect before re-open it?

[Cartucho]

You can leave it open and then we squash the commits before merging. This way I can also help you out.

[vuthede]

Ok, i see, thanks. Btw, I am working on:

• Fix bug.
• Make it works with more classes.
• Comment code.

[vuthede]

@Cartucho I have just finished:

• Fix some bugs,
• Make it works with more class. However, classIds I used is based on objectIds of object detector, which is different than "class_list.txt". If this feature is useful, I think we need a way to combine them.
• Comment code Would you mind to take a look on that? Thank you

[Cartucho]

I will have a look at that! Btw, we should compress the people_walking.mp4 video, using this website for example.

[Cartucho]

I didn't get a chance to look into this. However, this is by the far the most interesting PR that we have here!

Btw, what model are you using?

It would be nice to make this in a way to allow multiple models, we could have a folder called main/automatic_labeling and there a folder per model containing a README and a main_auto.py.

I also want to add a model for YOLOv3 and Mask R-CNN.

So it could look like this:     main/          input/          output/          automatic_labeling/              YOLOv3/              Mask_R-CNN/

[vuthede]

I have used the pretrained models on that link https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md. and it includes also Mask_R-CNN At the moment, i use only one file main_auto.py for using multiple model. In particular, I made the way so that we just change the path of pretrained model in file main_auto.py to use different model. What do u think?

[Cartucho]

Wow, I didn't know! That sounds great, let's use a single file then!

If they choose the Mask R-CNN however we would need to be able to deal with pixel labeling (segmentation).

I will set-up Tensor Flow so that I can test this then.

[Cartucho]

I'm having some problems in trying to find a computer with GPU to test this haha

[vuthede]

Haha, I made a config file, which currently allow using CPU by default. We can download the "light" model ssdlite_mobilenet_v2_coco_2018_05_09 on https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md and testing the logic of the code(we can add heavier model later). It work quite ok on CPU i think. At the moment, if the people using Mask R-CNN, the code still work ok with "bbox laleling". I haven't added pixel labeling as an option.

[Cartucho]

I will give it a try then!

[Cartucho]

I am getting this error:

[vuthede]

Ah sorry for no explanation Could u follow the instructions on Read me.

[vuthede]

@Cartucho I think it is because u may unintentionally use python 2.x. Python2x and Python3x may have different ways for importing modules. Could you use python3.x by type python3 main_auto.py instead of python main_auto.py

[Cartucho]

Hello @vuthede ! I owe you an apology for not responding any sooner. I have been moving out of my country since I just started a PhD in UK. I think in April I will have time to merge this PR!

[vuthede]

@Cartucho Yeah no problem. Wish you have a good time on UKI

[lstappen]

Hi! when is the merge planned? Do you need any help for testing? Might be interesting to extend this functionality to an active learning component (update the model after every new label) and let it run in parallel.

[Cartucho]

@lstappen yes, do you think you can help testing it out?

[Cartucho]

@vuthede can you help me break this down file by file?

Let's start simple. For example, you are adding a people walking video. I think that is a great idea but if do that we should remove the video of the rabbit. Do you agree? You only need one video as a sample

[vuthede]

Yeah I think It is ok to keep rabbit or walking video. I am lean abit toward using people walking video because I personally think it may be more popular, easier and practical to label people than anything else. And it seems like there is no rabbit class on object detection. I am very glad to help u, but could u show me break this down file by side means?

[Cartucho]

@vuthede it is essentially what we are doing now.

Ok, I will ask you to:

Remove rabbit video

cd main/input
git rm video.mp4

and add modify the following line in .gitignore:

from: main/input/video_mp4

to: main/input/people_walking_mp4

Change the file class_list.txt

from:

human
billiard ball
donut
rabbit

to

person
billiard ball
donut

[vuthede]

Yeah! Done

[Cartucho]

ok, now I think that release_output.sh is not the best name, you suggest changing it to remove_output.sh:

cd main
git mv release_output.sh remove_output.sh

[Cartucho]

The file object_detection/__pycache__/tf_object_detection.cpython-36.pyc should not be here. You should solve it by adding the following line *.pyc to the root .gitignore

[Cartucho]

Great! There are 5 files left to be reviewed:

• [ ] main/config.ini
• [ ] main/main_auto.py
• [ ] object_detection/tf_object_detection.py
• [ ] object_detection/utils.py
• [ ] README.md

Relatively to main/config.ini. What are the OBJECT_IDS = 1,2 and CUDA_VISIBLE_DEVICES = '' standing for?

[vuthede]

• OBJECT_IDS = 1,2 that means the object detector will return only object which having objectId = 1(people) or 2. I believe that those objects ID are Ids used in COCO dataset,which are different than the classlist.txt we use
• CUDA_VISIBLE_DEVICES = ''" means using CPU. If there is GPU on our machine, we can define device like CUDA_VISIBLE_DEVICES = ''0" to use GPU 0

[Cartucho]

Ok, I still don't understand completely how the OBJECT_ID works, but I will have a look into your code to find out.

[vuthede]

thanks!

[Cartucho]

You also need to add tensorflow as one of the prerequisites in the README.

[Cartucho]

We don't really need the config.ini file since we are using argsparse.

The main_auto.py and main.py should be merged together and then one of the parameters in the argsparse should define whether or not the user wants to use the automatic labeling. For example, the default value could be False but the user could activate it to True.

What do you think?

Can I commit changes to the code too?

[vuthede]

yeah absolutely. You can change the code and commit it. Yeah I think it is a good idea to use only main.py and using argsparse

[Cartucho]

Hey, I will be merging it as is. It is a great feature congratulations! :+1:

There is still some work to be done. Issues to be solved:

• [ ] the video is too big, needs to be cut, max 100 frames I would say
• [ ] the code should test if the model was indeed downloaded or not
• [ ] python3 and tensorflow are now a requirement. We should create a Conda environment
• [ ] does the code automatically tune to GPU vs. non-GPU usage?
• [ ] what happens if one runs the code twice? repeated labels?
• [ ] repeated code between main.py and main_auto.py
• [ ] remove config.ini and make the code use the classes in class_list.txt
• [ ] clean README it looks confusing now

These are the issues I plan to work on.