Multi Threading of the findObject object / Performances

[ghost]

Hi Mathieu,

Hope you are doing well !

I created another tool (httpRequest) for find object based on Tufao web server (https://github.com/vinipsmaker/tufao/tree/0.x, as Find-Object is using QT 4.8) and able to parse the request body from a POST request and forward it to the findObject server on the TCP Server:

1. Server Start (Port 1979): ./find_object --config ../config/find_object.ini --objects ../scene/dataset_1 --console --debug &
2. Webservice Start (include the tcpResponse class and create a connection to the port 1979): curl -s -X POST -H "Content-Type: multipart/form-data" -F "file=@test.jpg" http://localhost:8080/findObject | jq .

Output: { "Status": "0", "Info": [ "OpenCV version : 3.0.0-dev", "Major version : 3", "Minor version : 0", "Subminor version : 0", "getNumberOfCPUs: 40", "getNumThreads: 8", "getThreadNum: -667818048", "getTickFrequency: 1e+09", "getCPUTickCount: 7389729757512303", "useOptimized? 1", "Scene already having one channel and format CV_8U, (1 ms)" ], "Message": [ "No objects detected. (2 ms)" ] }

What it does ? The webservice is using several image pre-processing in order to extract exif informations if it is a JPEG, resize, crop a Region of Interest, finding blobs or squares :-)

I wanted to increase the number of requests per seconds for the web-service but we cannot have more than a request per port. So, I was wondering how we could multi thread or shard the findObject object created (https://github.com/introlab/find-object/blob/master/app/main.cpp#L463).

Why ? In order to increase performances and the size of the inverted vocabulary. I was thinking to use nanomsg to create a recursive tree of port to bind this service (either in parallel or recursively) but need to figure out how to scale the number of requests by binding several ports with a sort of internal load balancer with several copies of the vocabulary.

1- Is there a way to pass a parameters to the TCP socket to create a Q_SLOT in order to spawn a findObject children copy of any findObject object (if we can create several) but with a fixed size (like for the yaml vocabulary loading). Or to make the findObject object shared ?

2- I have often a "terminate called after throwing an instance of 'std::out_of_range" when I compute the homography in Find-object with the latest version. Would it be due to some parameters in the config.ini ?

3- Do you have any thoughts about how to multi-thread or create between 1 to 5 workers for the findObject app service without being too much intrusive ?

4- Have you seen the repo DBow2 providing an Enhanced hierarchical bag-of-word (https://github.com/thierrymalon/DBoW2/tree/sift) ? It implements a hierarchical tree for approximating nearest neighbours in the image feature space and creating a visual vocabulary. (using Dlib for the Machine Learning)

5- Did you have Rootsift (http://www.pyimagesearch.com/2015/04/13/implementing-rootsift-in-python-and-opencv/), Colour Feature Descriptors (https://github.com/eokeeffe/quasi_invariant-features), or MODS (http://cmp.felk.cvut.cz/wbs/index.html) on your roadmap for FindObject ?

6-. How complicated it would be to migrate to QT5 ^^ ?

Ps. I will make a cool web service :-) Happy Thanks Giving

Cheers, Luc Michalski

[ghost]

For the point #2, I have the following logs while compiling:

https://github.com/introlab/find-object/blob/master/src/FindObject.cpp#L377

In file included from /src/app/find-object/src/FindObject.cpp:13:0: /src/app/find-object/src/FindObject.cpp: In function 'void find_object::limitKeypoints(std::vectorcv::KeyPoint&, cv::Mat&, int)': /src/app/find-object/src/FindObject.cpp:362:27: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] UASSERT(keypoints.size() == descriptors.rows); ^ /src/app/find-object/src/../include/find_object/utilite/ULogger.h:66:33: note: in definition of macro 'UASSERT'

define UASSERT(condition) if(!(condition)) ULogger::write(ULogger::kFatal, FILE, LINE, FUNCTION, "Condition (%s) not met!", #condition)

                             ^

[matlabbe]

Hello Luc,

I updated the code for some of your questions:

1. I added "--tcp_threads" argument (default 1) to create TCP servers on multiple ports:

   ### Start the server
   ./find_object --console --object box.png --General/port 6000 --tcp_threads 4
   ...
   [ INFO] TcpServer set on port: 6000 (IP=192.168.0.100)
   [ INFO] TcpServer set on port: 6001 (IP=192.168.0.100)
   [ INFO] TcpServer set on port: 6002 (IP=192.168.0.100)
   [ INFO] TcpServer set on port: 6003 (IP=192.168.0.100)
   
   ### Then you can call all these simultaneously:
   ./find_object-tcpRequest --scene A.png --port 6000 &
   ./find_object-tcpRequest --scene B.png --port 6001 &
   ./find_object-tcpRequest --scene C.png --port 6002 &
   ./find_object-tcpRequest --scene D.png --port 6003

   If you add/remove objects (with "find_object-tcpService"), this will block all other calls (updating the same memory/vocabulary cannot be done in parallel).

2. Can you run find_object with "--debug" argument to have more info just before the crash (so we can more easily find which std::map or std::vector is accessed with a bad index)?
3. See 1.
4. I didn't know there was now an open source code for DBoW. I'll take a look.
5. Well, adding new features can be cool to compare. I opened an issue to add detectors/descriptors suggestions.
6. I updated the code, it can now be built under Qt5. Use cmake -DFINDOBJECT_QT_VERSION=5 .. to build with it.

Happy Thanks Giving you too! Mathieu

[matlabbe]

2 . This commit https://github.com/introlab/find-object/commit/c6ee7a27ef343ea782986adc4df1e86ebb8c999a would fix the problem.

[ghost]

I need to submit a dockerfile to your repo, did u had any chance to look at colorsift, or rootsift ?

2 interesting articles for u :-) http://www.computervisionblog.com/2016/01/why-slam-matters-future-of-real-time.html http://www.computervisionblog.com/2015/12/iccv-2015-twenty-one-hottest-research.html

[matlabbe]

I added rootsift in this commit. I didn't take a look at colorsift (do you mean this?).

Thx for the articles, very interesting!

[ghost]

Thanks a lot ! I recommend u to read tomasz, he is the king of computer vision :-)

[ghost]

Maybe another interesting poster related to your field: openvoc.berkeleyvision.org https://github.com/ronghanghu/open-voc-sergio/tree/openvoc-exp

Thanks again for rootsift, yes for colorsift I was thinking about the same repo you had.

[gitunit]

@lucmichalski openvoc is for object recognition?

[ghost]

@gitunit yes but the code is pretty messy with a deprecated dependency to the freebase api (knowledge graph dataset, stopped in june 2015). Also, caffe the framework has changed a lot since the latest commit of openvoc.

But here is the workflow suggested: http://openvoc.berkeleyvision.org/Open-Vocabulary_Object_Retrieval_RSS-2014-Poster.pdf

category matching > semantic segmentation of the scene (query expansion) > instance matching

[gitunit]

@lucmichalski ok, thx for the info. what is the alternative to freebase? google's knowledge graph?

[ghost]

@gitunit Dbpedia I would say but check this version of the Freebase RDF dump; http://basekb.com/gold/

[gitunit]

@lucmichalski thx. rootsift is great. i've determined already that it's superior to normal SIFT as some small objects were never detected before and now they are! :)

[ghost]

All credits should go to Mathieu :-) he is awesome

[gitunit]

@lucmichalski of course! but you deserve also some :smirk: big ups to @matlabbe it's me Walt btw :clap:

[matlabbe]

I'm glad that this change can do a little difference. Thx for the feedback!