search

abhineet123 / MTF

Modular Tracking Framework
http://webdocs.cs.ualberta.ca/~vis/mtf/
BSD 3-Clause "New" or "Revised" License
100 stars 23 forks source link
cpp decomposition homography image-registration mtf paper ros thesis tracking

readme

Welcome to the home of Modular Tracking Framework (MTF) !

Introduction

MTF is a highly efficient and extensible library for registration based tracking, also called direct visual tracking. It utilizes a modular decomposition of trackers in this domain wherein each tracker comprises the following 3 modules:

  1. Search Method (SM): ESM, IC, IA, FC, FA, LM, NN, PF, RANSAC or LMS
  2. Appearance Model (AM): SSD, SAD, ZNCC, SCV, LSCV, NCC, MI, CCRE, KLD, SSIM, SPSS, RIU, NGF, PCA or DFM
    • Illumination Model (ILM): GB, PGB and RBF
  3. State Space Model (SSM): Spline (50+ DOF), TPS (50+ DOF), Homography (8 DOF), Affine (6 DOF), ASRT (Anisotropic Scaling, Rotation and Translation - 5 DOF ), Similitude (4 DOF), AST (Anisotropic Scaling and Translation - 4 DOF), Isometry (3 DOF), IST (Isotropic Scaling and Translation - 3 DOF ) or Translation (2 DOF)

Documentation

Please refer these papers: [cviu] [iros17] for details on the system design, these: [crv16][wacv17][crv17] for some performance results and this thesis for a comprehensive description. There is also a dedicated website where Doxygen documentation will soon be available along with detailed tutorials and examples. It also provides several datasets formatted to work with MTF.

Implementation

The library is implemented entirely in C++ though interfaces for Python and MATLAB are also provided to aid its use in research applications. A simple interface for ROS is likewise provided for seamless integration with robotics projects. In addition to the registration tracking modules, MTF comes bundled with several state of the art learning and detection based trackers whose C++ implementations are publicly available - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT. It can thus be used as a general purpose tracking test bed too.

Platforms

MTF supports both Unix and Windows platforms. Though the Unix build system has been tested comprehensively only under Linux, specifically Ubuntu 14.04/16.04, it should work on Macintosh systems too (see Installation Issues section below for resolving possible issues). The Windows build system has been tested on Windows 8.1 and 10 with Visual Studio 2015, 2017 and 2019 though it should work fine with any non-ancient versions of the OS and IDE.

License and citing

MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper [bibtex] [arxiv version] be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it. Finally, if any issues are encountered while installing or running the library, please create an entry in the issues section and we will do our best to resolve it as soon as possible.

Prerequisites

General

Required

Optional

Windows

CMake with Visual Studio (Recommended)

Required

GNU Make with MinGW

Installation

Download the source code as zip file or clone using:
git clone https://github.com/abhineet123/MTF.git

Recommended

MTF comes with both a make and a cmake build system but the latter is recommended for most users of the library and can be used as follows:

The make system might be used by developers/contributors as it can potentially offer a finer level of control. Note, however, that it is not tested as well as the cmake system so is not guaranteed to work in all configurations or be up to date with the latest changes.

Advanced

Following are commands and switches for both make and cmake systems to customize the installation by disabling parts of the library or to only compile specific components. It also contains general information about the Python and Matlab wrappers and example applications. Not all of the make commands may work with the cmake system.

Commands

Examples

Python

make py/make install_py (make py2/make install_py2) : compile/install the Python interface to MTF (or its multi threaded version) - this creates a Python module called pyMTF (pyMTF2) that serves as a front end for using MTF from Python.

MATLAB

make mex (make mex2) : compile the MATLAB interface to MTF (or its multi threaded version) - this creates a MATLAB module called mexMTF (mexMTF2) that serves as a front end for using MTF from MATLAB.

Compile time switches

These apply to all of the above commands and the equivalent cmake options, where such exist, are given within parenthesis:

Clean up

Installation Issues

Cmake

Windows

Using Visual Studio 2017 or newer to build OpenCV 3.x.x might lead to a cmake error like Found OpenCV Windows Pack but it has not binaries compatible with your configuration. This can be resolved using the following steps which are for OpenCV 3.4.1 64 bit build with Visual Studio 2019 but adapting for other versions should be straightforward.

  1. Modify OpenCVConfig.cmake to add

    elseif(MSVC_VERSION MATCHES "^192[0-9]$")
    set(OpenCV_RUNTIME vc16)

right after

    elseif(MSVC_VERSION EQUAL 1900)
      set(OpenCV_RUNTIME vc14)
    elseif(MSVC_VERSION MATCHES "^191[0-9]$")
      set(OpenCV_RUNTIME vc15)
  1. Move bin, lib and include folders to x64/vc16/ after creating the same.

Compile time

Runtime

Macintosh

OpenCV 3.x

make build system

Setting Parameters

MTF parameters can be specified either in the cfg files present in the Config sub folder or from the command line. Please refer the ReadMe in the Config sub folder or this configuration page for detailed instructions.

Some preconfigured cfg files are provided here to get the system up and running quickly. Config files involving dataset sequences assume that all datasets are located in the Datasets sub folder under the active directory. The zip file should be extracted in the Config folder to replace the existing cfg files there (if any):

Running the demo application

Use either make run or runMTF to start the tracking application using the settings specified as above.

Building ROS package

A simple ROS package called mtf_bridge that demonstrates the integration of MTF with ROS applications is included in the ROS sub directory. Details about building and using this package are in the ReadMe in this folder.

Building a new application

cmake

The build process generates a mtfConfig.cmake file in the build folder with the main defines. There are two ways to use this file:

An example CMakeLists.txt file for a standalone project that uses either of the above methods is included in cmake/CMakeLists.txt.example.

If neither of the above methods work, a manually created FindMTF.cmake file is also included in the cmake/Modules folder. It has not been widely tested but should work with most standard configurations.

Windows

If MTF needs to be used within an existing VS project that was built without using cmake, the required preprocessor definitions and linked libraries can be added manually.

make

Use the make app app=<APPLICATION_NAME> command as detailed in the make commands section.

For Developers

Adding a new AM or SSM

cmake

  1. Modify the .cmake files in the respective sub directories:

    • Modify the AM/AM.cmake file to add the name of the AM to set(APPEARANCE_MODELS …)
    • Modify the SSM/SSM.cmake file to add the name of the SSM to set(STATE_SPACE_MODELS …)

  2. Modify "mtf.h" to add a new AM/SSM option in the overloaded function getTrackerObj (starting lines 498 and 533 respectively) that can be used with your config file to pick the AM/SSM, and create an object with this selected model.

  3. Modify "Macros/register.h" to add the new AM/SSM class name under _REGISTER_TRACKERS_AM/_REGISTER_HTRACKERS_AM or _REGISTER_TRACKERS/_REGISTER_TRACKERS_SSM/_REGISTER_HTRACKERS/_REGISTER_HTRACKERS_SSM respectively

make

  1. Modify the .mak file in the respective sub directory to:
    • add the name of the AM/SSM to the variable APPEARANCE_MODELS or STATE_SPACE_MODELS respectively
    • add rule to compile the .o of the new AM/SSM that is dependent on its source and header files - simplest method would be to copy an existing command and change the names of the model.
  2. Modify makefile to add any extra dependencies that the new files need to include under variable FLAGS64 and extra libraries to link against under LIB_MTF_LIBS

All steps are identical for adding a new Search Method (SM) too except the last one which is not needed. Instead this header needs to be included and the appropriate macro needs to be called from its source file to register the SM with all existing AMs and SSMs. Refer to the last 2 lines of the .cc file of any existing SM to see how to do this.

Examples

Implementing minimalistic AM that can be used with Nearest Neighbour SM

You need to create a new derived class from AppearanceModel. Implement the following functions:

  1. initializePixVals/updatePixVals : takes the current location of the object as a 2xN matrix where N = no. of sampled points and each column contains the x,y coordinates of one point. By default, it is simply supposed to extract the pixel values at these locations from the current image but there is no restriction by design and you are free to compute anything from these pixel values.
  2. initializeDistFeat/updateDistFeat : these compute a distance feature transform from the current patch.
  3. getDistFeat: returns a pointer to an array containing the distance feature vector computed by the above function. There is also an overloaded variant of updateDistFeat that takes a pointer as input and directly writes this feature vector to the pre-allocated array pointed to by this pointer.
  4. distance functor (operator()): computes a scalar that measures the dissimilarity or distance between two feature vectors (obtained using the previous two functions).

Implementing a minimalistic AM that can be used with Particle Filter SM

You need to create a new derived class from AppearanceModel. Implement the following functions:

  1. initializePixVals/updatePixVals : Same as with NN. No need to overwrite if you just need to extract the pixel values
  2. getLikelihood: get the likelihood/weights of the particle
  3. initializeSimilarity: initialize some class variables like the initial features.
  4. updateSimilarity: update the similarity between current patch and the template