Welcome to the home of Modular Tracking Framework (MTF) !
MTF is a highly efficient and extensible library for registration based tracking, also called direct visual tracking. It utilizes a modular decomposition of trackers in this domain wherein each tracker comprises the following 3 modules:
Please refer these papers: [cviu] [iros17] for details on the system design, these: [crv16][wacv17][crv17] for some performance results and this thesis for a comprehensive description. There is also a dedicated website where Doxygen documentation will soon be available along with detailed tutorials and examples. It also provides several datasets formatted to work with MTF.
The library is implemented entirely in C++ though interfaces for Python and MATLAB are also provided to aid its use in research applications. A simple interface for ROS is likewise provided for seamless integration with robotics projects. In addition to the registration tracking modules, MTF comes bundled with several state of the art learning and detection based trackers whose C++ implementations are publicly available - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT. It can thus be used as a general purpose tracking test bed too.
MTF supports both Unix and Windows platforms. Though the Unix build system has been tested comprehensively only under Linux, specifically Ubuntu 14.04/16.04, it should work on Macintosh systems too (see Installation Issues section below for resolving possible issues). The Windows build system has been tested on Windows 8.1 and 10 with Visual Studio 2015, 2017 and 2019 though it should work fine with any non-ancient versions of the OS and IDE.
MTF is provided under BSD license and so is free for research and commercial applications. We do request, however, that this paper [bibtex] [arxiv version] be cited by any publications resulting from projects that use MTF so more people can get to know about and benefit from it. Finally, if any issues are encountered while installing or running the library, please create an entry in the issues section and we will do our best to resolve it as soon as possible.
echo "export C_INCLUDE_PATH=${C_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc
echo "export CPLUS_INCLUDE_PATH=${CPLUS_INCLUDE_PATH}:/usr/include/eigen3" >> ~/.bashrc
nn=0
switch if FLANN is not available (see compile time switches).Install a recent version of Visual Studio
OpenCVConfig.cmake
file that comes with the latest version of OpenCV simply does not recognize Visual C++ 2017 so cmake might not be able to find it and instead exit with the erroneous message: Found OpenCV Windows Pack but it has not binaries compatible with your configuration
; a similar issue is exhibited by ViSP so it is best to use 2015 or older versions of Visual Studio; a possible way to resolve this issue is in the Installation Issues section if a newer version must be used. Set EIGEN3_ROOT
, EIGEN3_DIR
and EIGEN3_ROOT_DIR
environment variables to the folder containing the Eigen header files
Eigen
and unsupported
and a file called signature_of_eigen3_matrix_library
EIGEN_INCLUDE_DIR_WIN
variable in CMakeLists.txt
(line 13) to the location of this folderBuild OpenCV from source using Visual Studio or download the pre built binaries built with the correct version of visual studio if available.
-DWITH_THIRD_PARTY=0
to cmakeOpenCV_DIR
to the folder where the binaries have been built or downloaded; this should contain the OpenCVConfig.cmake
file along with the lib
and bin
foldersbin
folder to the Path
environment variableOPENCV_INCLUDE_DIR_WIN
, OPENCV_LIB_DIR_WIN
and OpenCV_SUFFIX
variables in CMakeLists.txt
(line 14-16) to the respective valuesBuild Boost from source using Visual Studio (toolset=msvc
) or download prebuilt binaries built with the correct version of visual studio if available.
BOOST_INCLUDEDIR
and BOOST_LIBRARYDIR
environment variables to the locations of the header and library (*.lib) files respectively.bin
folder containing the *.dll files to the Path
environment variableBOOST_INCLUDE_DIR_WIN
, BOOST_LIB_DIR_WIN
and Boost_SUFFIX
variables in CMakeLists.txt
(line 17-19) to the respective values .cmake
files and names of all .lib
and .dll
files to change the version number as needed.If FLANN is needed, build it from source using Visual Studio and set FLANN_ROOT
environment variable to the folder containing the compiled library and header files respectively
lib
, bin
and include
that contain the .lib, .dll and the header files bin
sub folder to the Path
environment variableIf ViSP is needed, build ViSP from source using Visual Studio and set VISP_DIR
environment variable to the folder containing the library and header files
VISPConfig.cmake
along with sub folders called x86/vc<version>/lib
, x86/vc<version>/bin
and include
that contain the .lib, .dll and the header files respectively.bin
folder to the Path
environment variable if the Python interface is to be built, install Python 2.7.x and Numpy.
MATLAB interface
Matlab_ROOT_DIR
or Matlab_ROOT_DIR_DEFAULT
in Examples.cmake to the MATLAB installation folder to help CMake find the correct version<Matlab_ROOT_DIR>/toolbox/local
by default; this location may need to be added to the MATLAB path for it to be found;ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.
due to an obscure incompatibility issue between the python import system and third party modules compiled with MinGW.OPENCV_INCLUDE_DIRS
and OPENCV_LIB_DIRS
in the makefile to the locations of header and library files respectively. toolset=gcc
) and set BOOST_LIB_DIRS
and BOOST_INCLUDE_DIRS
in the makefile to the locations of the header and library (*.dll.a
) files respectively.EIGEN_INCLUDE_DIRS
in the makefile to the location containing Eigen header files. BOOST_LIBS_SUFFIX
, and OPENCV_LIBS_SUFFIX
based on the installed versions of the respective libraries.PYTHON_INCLUDE_DIR
, PYTHON_LIBS_DIR
, NUMPY_INCLUDE_DIR
and MTF_PY_INSTALL_DIR
in Examples/Examples.mak
.bin
folders of the MinGW and GnuWin32 installations (e.g. C:\MinGW\bin
, C:\Program Files (x86)\GnuWin32\bin
) along with folders containing the *.dll
files of OpenCV and Boost installations (e.g. C:\OpenCV\build\x86\mingw\bin
) to the PATH
environment variable and reboot the system for the changes to take effect.Download the source code as zip file or clone using:
git clone https://github.com/abhineet123/MTF.git
MTF comes with both a make and a cmake build system but the latter is recommended for most users of the library and can be used as follows:
mkdir build && cd build && cmake ..
) to create the makefile for Unix or Visual Studio solution for Windowsmake
to compile (make -j<n>
for n-threaded compilation) and sudo make install
to installMTF.sln
in Visual Studio and run Build->Build Solution
Release
from Debug
to build the faster versionSolution Explorer->INSTALL
and select Build
to install the library, header and executable files to the installation directory (C:/MTF
by default); add <installation_folder>/bin
to Path
environment variable to be able to run the executables from any location;The make system might be used by developers/contributors as it can potentially offer a finer level of control. Note, however, that it is not tested as well as the cmake system so is not guaranteed to work in all configurations or be up to date with the latest changes.
Following are commands and switches for both make and cmake systems to customize the installation by disabling parts of the library or to only compile specific components. It also contains general information about the Python and Matlab wrappers and example applications. Not all of the make commands may work with the cmake system.
make
or make mtf
: compiles the shared library (libmtf.so) to the build directory (Build/Release)make install
: compiles the shared library if needed and copies it to /usr/local/lib; also copies the headers to /usr/local/include; (use make install_lib
or make install_header
for only one of the two); if third party trackers are enabled, their respective library files will be installed too;
MTF_INSTALL_DIR
in the makefile can be modified to install elsewhere. This can be done either by editing the file itself or providing it with the make/cmake command as:make install MTF_INSTALL_DIR=<installation_dir>
cmake .. -D MTF_INSTALL_DIR=<installation_dir>
lib
and include
sub folders within this folder should be present in LD_LIBRARY_PATH and C_INCLUDE_PATH/CPLUS_INCLUDE_PATH environment variables respectively so any application using MTF can find the library and headers.make exe
/make install_exe
: compile/install the main example file Examples/cpp/runMTF.cc to create an executable called runMTF
that uses this library to track objects.
MTF_INSTALL_DIR
as above if this is not availablemake mtfi
: all of the abovemake uav
/make install_uav
: compile/install an application called trackUAVTrajectory
that tracks the trajectory of a UAV in a satellite image of the area over which it flew while capturing images from abovemake mos
/make install_mos
: compile/install an application called createMosaic
that constructs a live mosaic from a video of the region to be stitchedmake qr
/make install_qr
: compile/install an application called trackMarkers
that automatically detects one or more markers in the input stream and starts tracking them
Data/Markers
folder by default; this can be changed by adjusting qr_root_dir
in Config/examples.cfg
where the names of marker files and the number of markers can also be specified along with some other parameters;make all
/make install_all
: compile/install all example applications that come with MTF along with the Python and Matlab interfacesmake app app=<APPLICATION_NAME>
: build a custom application that uses MTF with its source code located in <APPLICATION_NAME>.cc
; the compiled executable goes in the build directory;
MTF_APP_SRC_DIR
(defaults to Examples/cpp)make mtfa
will install it too - installation location can be specified through MTF_APP_INSTALL_DIR
(defaults to the current folder)make py
/make install_py
(make py2
/make install_py2
) : compile/install the Python interface to MTF (or its multi threaded version) - this creates a Python module called pyMTF (pyMTF2) that serves as a front end for using MTF from Python.
PY_VER
/py_ver
cmake/make variable (set to 2.7 by default) (e.g. -D PY_VER=3.5
for Python 3.5)PY_VER
is provided; in this case Python header and library locations must be provided though the variables PYTHON_INCLUDE_DIR
and PYTHON_LIBRARY
(e.g. -D PYTHON_INCLUDE_DIR=/usr/include/python3.5 -D PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so
for Python 3.5)MTF_PY_INSTALL_DIR
(in Examples.mak/cmake) if using a custom Python version (defaults to /usr/local/lib/python2.7/dist-packages/)python setup.py install
or python3 setup.py install
from Examples/python folder.
Its usage is demonstrated in the ReadMe in Examples folder.make mex
(make mex2
) : compile the MATLAB interface to MTF (or its multi threaded version) - this creates a MATLAB module called mexMTF (mexMTF2) that serves as a front end for using MTF from MATLAB.
MATLAB_DIR
variable in Examples.mak to the root of the MATLAB installation folder MTF_MEX_INSTALL_DIR
in Examples.mak/cmake (defaults to _mexMTF
in Unix even if MATLAB is not detected by CMake by running the command written to a file called mtf_mex_cmd.txt
in the build folder;
mex
executable is present in the PATH
environment variable;These apply to all of the above commands and the equivalent cmake options, where such exist, are given within parenthesis:
only_nt=1
/nt=1
(WITH_TEMPLATED=OFF
) will enable only the Non Templated (NT) implementations of SMs and disable their templated versions that are extremely time consuming to compile though being faster at runtime (disabled by default)
nn=0
(WITH_FLANN=OFF
) will disable the templated implementation of NN search method (enabled by default).
spi=1
(WITH_SPI=ON
) will enable support for selective pixel integration in modules that support it (disabled by default)
grid=0
(WITH_GRID_TRACKERS=OFF
) will disable the Grid trackers and RKLT (enabled by default).feat=0
(WITH_FEAT=OFF
) will disable the Feature tracker (enabled by default).
lt=0
(WITH_THIRD_PARTY=OFF
) will disable the third party open source learning based trackers - DSST, KCF, CMT, TLD, RCT, MIL, Struck, FragTrack, GOTURN and DFT - that are also bundled with this library (in ThirdParty subfolder) (enabled by default except MIL, DFT and GOTURN).
gtrn=1
(WITH_GOTURN=ON
) will enable GOTURN deep learning based tracker (disabled by default)
~/caffe/build/install
, specify CAFFE_INSTALL_DIR
either at compile time or by editing ThirdParty/GOTURN/GOTURN.mak
dft=1
(WITH_DFT=ON
) will enable the Descriptor Fields Tracker (disabled by default)
mil=1
(WITH_MIL=ON
) will enable the Multiple Instance Learning tracker (disabled by default)
vp=1
(WITH_VISP=ON
) will enable ViSP template trackers and input pipeline (disabled by default).
LD_LIBRARY_PATH
variable by running, for instance, echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/x86_64-linux-gnu/" >> ~/.bashrc
xv=1
(WITH_XVISION=ON
) will enable Xvision trackers and input pipeline (disabled by default and not recommended).o=0
will compile the library in debug mode (_libmtfdebug.so) and disable optimization - the corresponding executables will be suffixed with 'd' ( for example runMTFd)ver=<version_postfix>
will postfix all compiled library and executable files with the provided postfix so that multiple versions may coexist on the same system without conflict; e.g. ver=2
will create libmtf_v2.so and runMTF_v2 and same for all libraries created during MTF compilation (including those for third party trackers);header_only=1
can be used with make exe
or make mtfe
to build a stand alone version of runMTF called runMTFh that does not need to link with libmtf.so
make clean
: removes all the .o files and the .so file created during compilation from the Build folder.make mtfc
: also removes the executable.Found ViSP for Windows but it has no binaries compatible with your configuration. You should manually point CMake variable VISP_DIR to your build of ViSP library.
occurs, add -DVISP_DIR=<path to ViSP build folder>
to the cmake command and rerun after removing the existing cmake filesUsing Visual Studio 2017 or newer to build OpenCV 3.x.x might lead to a cmake error like Found OpenCV Windows Pack but it has not binaries compatible with your configuration
.
This can be resolved using the following steps which are for OpenCV 3.4.1 64 bit build with Visual Studio 2019 but adapting for other versions should be straightforward.
Modify OpenCVConfig.cmake
to add
elseif(MSVC_VERSION MATCHES "^192[0-9]$")
set(OpenCV_RUNTIME vc16)
right after
elseif(MSVC_VERSION EQUAL 1900)
set(OpenCV_RUNTIME vc14)
elseif(MSVC_VERSION MATCHES "^191[0-9]$")
set(OpenCV_RUNTIME vc15)
x64/vc16/
after creating the same.if an error like cannot find -l<name of library>
(for instance cannot find -lhdf5
or cannot find -lmtf
) occurs at compile time, location of the corresponding library should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables in the .bashrc file.
echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/hdf5/lib" >> ~/.bashrc
echo "export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib" >> ~/.bashrc
. ~/.bashrc
and compilation should be tried again.similar compile time errors pertaining to missing header files may be encountered in which case the location of the include
folder of the corresponding library should be added to C_INCLUDE_PATH and CPLUS_INCLUDE_PATH variables. For instance, following commands can be run for HDF5:
echo "export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/hdf5/include" >> ~/.bashrc
echo "export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/usr/local/hdf5/include" >> ~/.bashrc
if a compile time error similar to this occurs: /usr/include/eigen3/Eigen/src/Core/util/BlasUtil.h:233:98: error: no matching function for call to ‘Eigen::internal::blas_traits
, please update Eigen to the latest version;
if a compile time error similar to this occurs:
/usr/bin/ld: cannot find -lQt5::Core
/usr/bin/ld: cannot find -lQt5::Gui
/usr/bin/ld: cannot find -lQt5::Widgets
/usr/bin/ld: cannot find -lQt5::Test
/usr/bin/ld: cannot find -lQt5::Concurrent
/usr/bin/ld: cannot find -lQt5::OpenGL
there is probably something wrong with the OpenCV installation. Reinstalling that should fix it (refer to issue #13).
A compile time error similar to this:
error: no matching function for call to vpDisplay::init(vpImage<vpRGBa>&, int, int, std::string&)
probably indicates that an outdated version of ViSP is installed. This can be resolved by either updating it to 3.0.1 or newer or using -DWITH_VISP=0
/vp=0
cmake/make option to disable ViSP.
A compile time error similar to this:
undefined reference to LZ4_compress_HC_continue
occurs when the cmake module for one of the dependencies is not configured properly and does not return one or more libraries it needs to be linked against (liblz4
in this case).
-D MTF_LIBS=<name_of_missing_library>
(in this case: -D MTF_LIBS=lz4
) to the cmake command.sudo apt-get install liblz4-dev
if a runtime error like cannot find libmtf
or something similar is encountered while running runMTF
, /usr/local/lib should be added to LIBRARY_PATH and LD_LIBRARY_PATH environment variables as above (assuming the default MTF installation directory has not been changed - otherwise the new path should be used instead).
if a run time error similar to this:
undefined symbol: _ZN3cmt7Matcher10initializeERKSt6vectorIN2cv6Point_IfEESaIS4_EENS2_3MatERKS1_IiSaIiEES9_S4_
suddenly occurs or running runMTF
causes segmentation fault with no messages right after installing ROS, it is probably because MTF is linking with an incorrect version of OpenCV installed by ROS.
lt=0
to disable all third party modules is the best option.export CC=/usr/bin/gcc
export CXX=/usr/bin/g++
export CMAKE_PREFIX_PATH=${CMAKE_PREFIX_PATH}:/Applications/HDF_Group/HDF5/1.10.0-patch1/
lt=0
/-DWITH_THIRD_PARTY=OFF
if compilation or linking errors pertaining to these are found./usr/bin/ld: cannot find -lippicv
occurs, remove -ippicv
from opencv pkg-config configuration file which is usually located at /usr/local/lib/pkgconfig/opencv.pc
or follow the procedures suggested hereonly_nt=0
(or nn=1
) and is recompiled with only_nt=1
(or nn=0
) such that the executable does not get recompiled, a runtime error of the type: symbol lookup error: runMTF: undefined symbol:
may occur on running the executable; to resolve this, remove runMTF
(or whatever executable is causing the error) from the build folder (Build/Release or Build/Debug) by running, for instance rm Build/Release/runMTF
and recompile; this happens because using only_nt=1
or nn=0
turns off part of the code in the library but the executable will continue to expect these to be present till it is recompiled;cmake .. -DWITH_THIRD_PARTY=OFF -DWITH_TEMPLATED=OFF
make mtfall lt=0 nt=1
-DWITH_FLANN=OFF
/nn=0
can be specified too -DWITH_FEAT=OFF
/feat=0
can be specified too MTF parameters can be specified either in the cfg files present in the Config sub folder or from the command line. Please refer the ReadMe in the Config sub folder or this configuration page for detailed instructions.
Some preconfigured cfg files are provided here to get the system up and running quickly. Config files involving dataset sequences assume that all datasets are located in the Datasets
sub folder under the active directory. The zip file should be extracted in the Config
folder to replace the existing cfg files there (if any):
Use either make run
or runMTF
to start the tracking application using the settings specified as above.
A simple ROS package called mtf_bridge
that demonstrates the integration of MTF with ROS applications is included in the ROS
sub directory.
Details about building and using this package are in the ReadMe in this folder.
The build process generates a mtfConfig.cmake
file in the build folder with the main defines.
There are two ways to use this file:
The standard find_package
command can be used but this requires MTF_DIR
variable to be set to the MTF build folder while running the cmake command.
This can be done, for instance, as:
cmake .. -DMTF_DIR=<path to MTF build folder>
The mtfConfig.cmake
file can be copied into the project tree and included in the CMakeLists.txt file. Then, the defined variables can be used to obtain the header files, linked libraries and compile time definitions. More details can be found in the comments section of issue #11.
An example CMakeLists.txt file for a standalone project that uses either of the above methods is included in cmake/CMakeLists.txt.example.
If neither of the above methods work, a manually created FindMTF.cmake
file is also included in the cmake/Modules
folder. It has not been widely tested but should work with most standard configurations.
If MTF needs to be used within an existing VS project that was built without using cmake, the required preprocessor definitions and linked libraries can be added manually.
Right click on one of the executables in MTF solution (e.g. runMTF
), go to Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions
and copy all flags there to the same location in your own project.
Repeat this with Properties -> Linker -> Input -> Additional Dependencies
to add all required libraries for linking.
Use the make app app=<APPLICATION_NAME>
command as detailed in the make commands section.
Modify the .cmake files in the respective sub directories:
Modify "mtf.h" to add a new AM/SSM option in the overloaded function getTrackerObj
(starting lines 498 and 533 respectively) that can be used with your config file to pick the AM/SSM, and create an object with this selected model.
Modify "Macros/register.h" to add the new AM/SSM class name under _REGISTER_TRACKERS_AM/_REGISTER_HTRACKERS_AM
or _REGISTER_TRACKERS/_REGISTER_TRACKERS_SSM/_REGISTER_HTRACKERS/_REGISTER_HTRACKERS_SSM
respectively
APPEARANCE_MODELS
or STATE_SPACE_MODELS
respectivelyFLAGS64
and extra libraries to link against under LIB_MTF_LIBS
All steps are identical for adding a new Search Method (SM) too except the last one which is not needed. Instead this header needs to be included and the appropriate macro needs to be called from its source file to register the SM with all existing AMs and SSMs. Refer to the last 2 lines of the .cc file of any existing SM to see how to do this.
You need to create a new derived class from AppearanceModel. Implement the following functions:
initializePixVals/updatePixVals
: takes the current location of the object as a 2xN matrix where N = no. of sampled points and each column contains the x,y coordinates of one point. By default, it is simply supposed to extract the pixel values at these locations from the current image but there is no restriction by design and you are free to compute anything from these pixel values.initializeDistFeat/updateDistFeat
: these compute a distance feature transform from the current patch.getDistFeat
: returns a pointer to an array containing the distance feature vector computed by the above function. There is also an overloaded variant of updateDistFeat that takes a pointer as input and directly writes this feature vector to the pre-allocated array pointed to by this pointer.operator()
): computes a scalar that measures the dissimilarity or distance between two feature vectors (obtained using the previous two functions).You need to create a new derived class from AppearanceModel. Implement the following functions:
initializePixVals/updatePixVals
: Same as with NN. No need to overwrite if you just need to extract the pixel valuesgetLikelihood
: get the likelihood/weights of the particleinitializeSimilarity
: initialize some class variables like the initial features.updateSimilarity
: update the similarity between current patch and the template