Memory leak when TCP request with many object in session

[nhudinh2103]

Hi matlabe,

When use find object with many object added (size >= 10000), tcpRequest will cause memory leak

Here's image:

1. Start find-object with existed session (object size = 10000)

2. After some request with find-object-tcpRequest

Things will get worse if I start you find object with tcpThreads = 4

I have find out this block of code cause memory leak. (If I comment code to stop Homorgraphy thread, find-object will stop leak).

FindObject.cpp:1587

QVector<HomographyThread*> threads;

UDEBUG("Creating/Starting homography threads (%d)...", threadCounts);
for(int k=i; k<i+threadCounts && k<matchesList.size(); ++k)
{
    int objectId = matchesId[k];
    UASSERT(objects_.contains(objectId));
    threads.push_back(new HomographyThread(
                            &matchesList[k],
                            objectId,
                            &objects_.value(objectId)->keypoints(),
                            &info.sceneKeypoints_,
                            objects_.value(objectId)->image(),
                            grayscaleImg));
    threads.back()->start();
}
    UDEBUG("Started homography threads");

    for(int j=0; j<threads.size(); ++j)
    {
        threads[j]->wait();
        ...

I try to comment code in run() method in HomographyThread class, it still leak. So I think there may be some problem with thread.

FindObject.cpp:1196

virtual void run()
    {
        //QTime time;
        //time.start();

        std::vector<cv::Point2f> mpts_1(matches_->size());
        std::vector<cv::Point2f> mpts_2(matches_->size());
        indexesA_.resize(matches_->size());
        indexesB_.resize(matches_->size());

        UDEBUG("Fill matches...");
        int j=0;
        for(QMultiMap<int, int>::const_iterator iter = matches_->begin(); iter!=matches_->end(); ++iter)
        {
            UASSERT_MSG(iter.key() < (int)kptsA_->size(), uFormat("key=%d size=%d", iter.key(),(int)kptsA_->size()).c_str());
            UASSERT_MSG(iter.value() < (int)kptsB_->size(), uFormat("key=%d size=%d", iter.value(),(int)kptsB_->size()).c_str());
            mpts_1[j] = kptsA_->at(iter.key()).pt;
            indexesA_[j] = iter.key();
            mpts_2[j] = kptsB_->at(iter.value()).pt;
            indexesB_[j] = iter.value();
            ++j;
        }

        if((int)mpts_1.size() >= Settings::getHomography_minimumInliers())
        {
            if(Settings::getHomography_opticalFlow())
            {
                UASSERT(!imageA_.empty() && !imageB_.empty());

                cv::Mat imageA = imageA_;
                cv::Mat imageB = imageB_;
                if(imageA_.cols < imageB_.cols && imageA_.rows < imageB_.rows)
                {
                    // padding, optical flow wants images of the same size
                    imageA = cv::Mat::zeros(imageB_.size(), imageA_.type());
                    imageA_.copyTo(imageA(cv::Rect(0,0,imageA_.cols, imageA_.rows)));
                }
                if(imageA.size() == imageB.size())
                {
                    UDEBUG("Optical flow...");
                    //refine matches
                    std::vector<unsigned char> status;
                    std::vector<float> err;
                    cv::calcOpticalFlowPyrLK(
                            imageA,
                            imageB_,
                            mpts_1,
                            mpts_2,
                            status,
                            err,
                            cv::Size(Settings::getHomography_opticalFlowWinSize(), Settings::getHomography_opticalFlowWinSize()),
                            Settings::getHomography_opticalFlowMaxLevel(),
                            cv::TermCriteria(cv::TermCriteria::COUNT+cv::TermCriteria::EPS, Settings::getHomography_opticalFlowIterations(), Settings::getHomography_opticalFlowEps()),
                            cv::OPTFLOW_LK_GET_MIN_EIGENVALS | cv::OPTFLOW_USE_INITIAL_FLOW, 1e-4);
                }
                else
                {
                    UERROR("Object's image should be less/equal size of the scene image to use Optical Flow.");
                }
            }

            UDEBUG("Find homography... begin");
#if CV_MAJOR_VERSION < 3
            h_ = findHomography(mpts_1,
                    mpts_2,
                    Settings::getHomographyMethod(),
                    Settings::getHomography_ransacReprojThr(),
                    outlierMask_);
#else
            h_ = findHomography(mpts_1,
                    mpts_2,
                    Settings::getHomographyMethod(),
                    Settings::getHomography_ransacReprojThr(),
                    outlierMask_,
                    Settings::getHomography_maxIterations(),
                    Settings::getHomography_confidence());
#endif
            UDEBUG("Find homography... end");

            UASSERT(outlierMask_.size() == 0 || outlierMask_.size() == mpts_1.size());
            for(unsigned int k=0; k<mpts_1.size();++k)
            {
                if(outlierMask_.size() && outlierMask_.at(k))
                {
                    inliers_.insert(indexesA_[k], indexesB_[k]);
                }
                else
                {
                    outliers_.insert(indexesA_[k], indexesB_[k]);
                }
            }

            if(inliers_.size() == (int)outlierMask_.size() && !h_.empty())
            {
                if(Settings::getHomography_ignoreWhenAllInliers() || cv::countNonZero(h_) < 1)
                {
                    // ignore homography when all features are inliers
                    h_ = cv::Mat();
                    code_ = DetectionInfo::kRejectedAllInliers;
                }
            }
        }
        else
        {
            code_ = DetectionInfo::kRejectedLowMatches;
        }

        //UINFO("Homography Object %d time=%d ms", objectIndex_, time.elapsed());
    }

Can you help me to solve it? Thanks

[matlabbe]

Looks similar to #55. Try updating to latest version.

Cheers, Mathieu

[nhudinh2103]

Hi Mathieu,

Thanks for your quick response

I have pulled lastest version and built it, but it's still leak memory

Here is what I did:

1. Run find_object in console, 4 tcp threads (after run RAM = 10.1GB)

./find_object --images_not_saved --console --session ./test_incremental.bin --tcp_threads 4

2. Run tcp_request, repeat 10 times (after run 10 times RAM = 10.8GB and increase for each request)

./find_object-tcpRequest --json "out.json" --scene ./screen1.jpg --port 6000

[matlabbe]

I did try something similar by making a session of these 2 objects and setting General/port to 6000, then: ./find_object --images_not_saved --console --session /Users/mathieu/test.bin --tcp_threads 4

Running this script (calling 1000 times tcpRequest):

#!/bin/bash
# Basic while loop
counter=1
while [ $counter -le 1000 ]
do
echo $counter
./find_object-tcpRequest --json "out.json" --scene /Users/mathieu/Documents/workspace/find-object/bin/multi-scene.jpg --port 6000
((counter++))
done
echo All done

It started and finished at 12.5 MB, sometimes increasing to 15 MB or 20 MB but decreasing back to 12.5 MB. I tested on Mac OS X. You may try your test for a more long time and see if it keeps increasing. It is sure that allocating/deallocating memory sometimes doesn't restore back to 0 (depending on the OS), but it should somewhat stabilize at some point. Otherwise, if you can reproduce the problem on a smaller session (my laptop has only 8GB RAM) and if you can share it, it may be easier to reproduce the problem.

I just tried adding 200 more objects to the session. It started and finished at 40 MB too.

cheers, Mathieu

[nhudinh2103]

Hi Matthieu,

Here's some information about my environment: OS: Linux (CentOS 7) RAM: 16GB

My find_object config: (I use SIFT detector/descriptor)

https://drive.google.com/open?id=1Ayvdaa5lh4hrcXLvX0QHSFv7z53I6Hms

Here's dataset you can load object for test (Sorry I can't share my own dataset, so I just use this to test)

https://drive.google.com/open?id=1l4CxTHH-qblPhMP6ZqxQFGHYenzY5a4d

Screen

I load objects from dataset above and run tcp request.

./find_object-tcpRequest --json "out.json" --scene ./screen1.jpg --port 6000

Request 10 times: increasing 0.2 GB (7.3GB -> 7.5GB) Request 20 times (after run 10 times): increasing 0.3 GB (7.5GB -> 7.8GB)

[matlabbe]

I tried why your data and on Windows and MAC OS X, the memory used by find-object stays around 200 MB (before and after 20 calls to tcpRequest). On Ubuntu, it started at 200 MB, but increased up to 895 after 15 times, then remained between 895 and 905 after 15 more times. I am not sure if it is how Ubuntu is computing the memory usage that may look like sometimes they don't deallocate all memory. We may need to try with some memory analysis tools to verify that.

[nhudinh2103]

Hi matlabe, thanks for your test and response

I'm currently use heaptrack to detect memory leak (because valgrind too slow and not show which line of code).

https://github.com/KDE/heaptrack

[nhudinh2103]

Hi matlabbe,

I finally got the answer for this issues.

It's because some problem with default malloc (glibc) in Linux. A workarround to this is use tcmalloc library from google.

https://github.com/introlab/find-object/pull/62

If you have a better way, let me know it.

Dinh

[matlabbe]

I merged the pull request, but I updated it so that tcmalloc dependency is not required.