search

tmandel / fish-detrac

18 stars 2 forks source link

readme

Welcome to the FISHTRAC codebase (and dataset)

Welcome to the FISHTRAC codebase! This includes a harness for comparing 17 Multi-Object Tracking (MOT) algorithms on three datasets, links to our new real-world fish tracking dataset, and the ability to easily run trackers on new datasets or detections. It also includes code for our new MOT algorithm (RCT aka KPD) that works well in domains like fish tracking where detection is imperfect, and the ability to easily run trackers on new datasets or detections.

If you use this codebase (or any part of it, such as the RCT tracker), please cite the following associated publication:

Detection Confidence Driven Multi-Object Tracking to Recover Reliable Tracks From Unreliable Detections

@article{mandel2023detection,
title = {Detection confidence driven multi-object tracking to recover reliable tracks from unreliable detections},
author = {Travis Mandel and Mark Jimenez and Emily Risley and Taishi Nammoto and Rebekka Williams and Max Panoff and Meynard Ballesteros and Bobbie Suarez},
journal = {Pattern Recognition},
volume = {135},
pages = {109107},
year = {2023},
issn = {0031-3203},
doi = {https://doi.org/10.1016/j.patcog.2022.109107},
url = {https://www.sciencedirect.com/science/article/pii/S0031320322005878}
}

Our test harness is a heavily modified and extended version of the UA-DETRAC toolkit.

There are a lot of MOT codebases out there. Why consider using ours?

Acknowledgement

In addition to the acknowledgements listed in the paper, the technical support and advanced computing resources from University of Hawaii Information Technology Services – Cyberinfrastructure, funded in part by the National Science Foundation MRI award # 1920304, are gratefully acknowledged.

Initial setup

This is designed to work only on Linux OS. The steps you should follow are:

  1. Clone the repo
  2. Install Anaconda
  3. Install GNU Octave (version 4.4.1+) sudo apt-get install octave && sudo apt-get install liboctave-dev (this will install version 5.2.0 on ubuntu 20, if you are ubuntu 18 or earlier we recommend installing version 5.2.0 or 6.2.0 from source )
  4. Install CMake
  5. (Only needed for TransCenter tracker) Install Singularity (we used version 3.7.2) [https://docs.sylabs.io/guides/3.5/admin-guide/installation.html]
  6. (Only needed for JPDA-M tracker) Install flatpak, and Octave 6.2.0 inside flatpak, which needs the io and statistics packages 
    sudo apt install flatpak
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
flatpak install flathub org.octave.Octave
sudo flatpak update --commit=51d145ea9ae2bd4bfb6d5518fd8d6c1833753d98d102ec9deae589dfea25d9b9 org.octave.Octave
sudo flatpak install flathub org.kde.Platform//5.15
sudo flatpak install flathub org.kde.Sdk//5.15
flatpak run org.octave.Octave
pkg install io-2.6.3.tar.gz
pkg install statistics-1.4.3.tar.gz
quit()
  7. Download fishtrac-extras.zip and place it in the outer fish-detrac directory (this contains large files that are too big to place in the repo).
  8. cd into the fish-detrac directory and run ./setupFiles.sh to create the necessary directories and conda environments, as well as download and setup the D3S tracker
  9. Follow the instructions printed at the end of setupFiles.sh to install the necessary Octave packages.

Running trackers on datasets

  1. Download your dataset(s) of choice and unzip it into the fishtrac folder. Our tutorial below will use FISHTRAC train, but the process is similar for the other choices.

Note: By downloading and using FISHTRAC data, you hereby agree not to use not use FISHTRAC data for harmful/unethical purposes, for instance better killing, harvesting, harming, or exploiting fish populations. FISHTRAC is meant to aid conservation of fish and non-invasive studies of fish behavior.

  1. Choose the sequence file that matches your dataset our interest:

We provide several sequence files in fishtrac/evaluation/seqs:

  1. To run trackers, we can use experiment wrapper. Experiment wrapper lets you specify the tracker, the detection confidence threshold, and the sequence file name (see step 9). It will automatically record MOTA and HOTA metrics and time out the trackers if they have not completed in 30 minutes. As an example, we will run GOG and RCT (called KPD in the code). We run GOG with a 0.3 threshold on detection confidence because that is what we found to perform best on our train data; the threshold for RCT is arbitrary: it does not use the threshold as it consumes unfiltered detections. 
    cd fishtrac
conda activate fish_env
python experiment_wrapper.py GOG 0.3 fishtrac-train octave-cli
python experiment_wrapper.py KPD 0.5 fishtrac-train octave-cli

Note that you might need to replace octave-cli part if you have a different way of running octave (e.g. a custom compiled version, etc.) If it succeeded, you should see a relatively short output and a message like: 

`run tracker command returned: 0`

otherwise, there may have been some error in the installation process.

  1. Next, we will take a look at some of the scores: 
    cd compare-trackers/
python benchmark_evaluation.py fishtrac-train GOG,KPD train
cat benchmark_results.csv

    Note : the train argument means we will run over all detection confidence thresholds that have been run to determine the best and store it off in a file in fishtrac/results/best_thresholds/. If we were to specify test instead of train, it would read and used the stored threshold rather than searching for the best.

Pasting the displayed info into a spreadsheet program will show that RCT (KPD) has a much better HOTA and MOTA than GOG on the FISHTRAC train videos. In fact, they should be essentially the same results as we provided for "Fish Vids" in the provided full_paper_results/Train_results.xlsx spreadsheet that we based our paper results on (along with full_paper_results/Test_results.xlsx).

  1. However, it is also useful to see qualitative performance (that is, what the trackers actually look like in action!). Thankfully, we also provide scripts for that. Here is an example of visualizing the methods on one of the three videos:
    cd ../convert_gt
    python mergeFrame.py ../DETRAC-images/02_Oct_18_Vid-3
    python showOnVideo.py ../DETRAC-images/02_Oct_18_Vid-3.mp4 GOG_Test.mp4 ../results/GOG/R-CNN/0.3/02_Oct_18_Vid-3_LX.txt GOG
    python showOnVideo.py ../DETRAC-images/02_Oct_18_Vid-3.mp4 RCT_Test.mp4 ../results/KPD/R-CNN/0.5/02_Oct_18_Vid-3_LX.txt RCT

Then view the GOG_Test.mp4 and RCT_Test.mp4 videos in your favorite video player! You are now done with the basic tutorial!

Tracker-specific notes

Using your own dataset:

The FISHTRAC codebase is intended to provide a general-purpose way to evaluate tracking-by-detection methods on a variety of datasets, including new ones that other researchers have collected. Tracking-by-detection methods typically have three sources of input: Raw videos (mp4 or folder of frames), detections, and annotations (so that the results can be rigorously evaluated).

Getting detections

The first step to getting detections would be to train a detector on your domain. For this, you will need some annotated image data.

But what if I have little to no annotated images data? You can follow the approach we took in the paper and download images from Google Open Images - see https://github.com/aferriss/openImageDownloader

In the paper we did a query like the following to grab all human-annotated bounding box data of fish (you will need to change the label name depending on your label of interest):

#standardsql
SELECT
  i.image_id AS image_id,
  original_url,thumbnail_300k_url,
  confidence
FROM
  `bigquery-public-data.open_images.annotations_bbox` l
INNER JOIN
  `bigquery-public-data.open_images.images` i
ON
  l.image_id = i.image_id
WHERE
  label_name='/m/0ch_cf'
  AND source='human'
  AND Subset='train'

Next, you will need to covert those annotations to covert these annotations to a format that you can train on. We have provided a script to convert to keras-retinanet format in fishtrac/convert_gt/detection_train/annotations.py

Next, you will need to train the detector over your annotations. We used https://github.com/fizyr/keras-retinanet (version 0.5.0) to train a retinanet model on the CSV file and then convert the training h5 file to an inference h5 file. Although this method is a bit old, it outperformed more recent approaches such as YOLOv4 on our dataset, as we explain in the paper.

Finally, you will need to run the detector over the video file(s) and create the resulting detection file. This is the purpose of fishtrac/convert_gt/outputDetectorPredictions.py. It should generate a Detection MAT file which can be placed in the DETRAC-Train-Detections/R-CNN folder.

From mp4 video and vatic.js style annotations

In order to evaluate trackers, we need video annotations. As an example of how we did this, in the raw_data folder (provided by fishtrac-extras) I have provided the raw mp4s and raw xml annotation files produced by the tool vatic.js for Fishtrac Train.

  1. Copy the files into convertGT:

    cp raw_data/*.mp4 fishtrac/convert_gt
cp raw_data/*.xml fishtrac/convert_gt

  2. Run convertLocal.py over the video to convert the annotation to DETRAC format.

    cd fishtrac/convert_gt
python convertLocal.py

    (When prompted, select the 02_Oct_18_Vid-3.mp4, then run again for the other two mp4 files)

    This creates numerous items:

    • A folder of images in DETRAC format for the video
    • The annotations in mat format
    • The annotations in a similar format to how DETRAC specifies tracks
    • A video to help you review the annotations and make sure they look correct

  3. Now, we need to move the files in the right place for the FISHTRAC experiment. 

    cp *.mat ../DETRAC-Train-Annotations-MAT/
cp -r 02_Oct_18_Vid-3 ../DETRAC-images/
cp -r V1_Leleiwi_26June19_17 ../DETRAC-images/
cp -r V1_Leleiwi_26June19_22 ../DETRAC-images/

Once you have placed the detections from the fishtrac-train.zip file in the DETRAC-Train-Detections/R-CNN, you should be able to run as normal and get the same results as we did when we just used the processed annotations and images from fishtrac-train.zip.

From MOT-17 frame images and MOT17 annotations

Some may have their images and annotations already in MOT17 format so as to participate in the MOTChallenges, etc.

We provide scripts for this in the fishtrac/convert_gt/MOT17convert folder. If you make a trainSet folder and put the MOT17 videos that end in DPM there, running convertMOTImages.sh will convert the images to DETRAC format. Running convertAllGT.sh will convert the MOT17 ground truth files to DETRAC format. Then you can place the converted images in the fishtrac/DETRAC-images/ folder and the ground truth mat files in the fishtrac/DETRAC-Train-Annotations-MAT/ folder.

License info

The FISHTRAC evalulation harness is a (heavily modifed) derivative work of the UA-DETRAC evaluation codebase. UA-DETRAC is also licensed CC BY-NC-SA 3.0. UA-DETRAC was developed by Longyin Wen, Dawei Du, Zhaowei Cai, Zhen Lei and Ming-Ching Chang, Honggang Qi, Jongwoo Lim, Ming-Hsuan Yang, Siwei Lyu, Yi Wei, Yuezun Li,Tao Hu, and Siwei Lyu.

Therefore, the FISHTRAC evaluation harness code (excluding the HOTA code and the code for the individual trackers) is also licensed CC BY-NC-SA 3.0. The FISHTRAC evaluation code was developed by Travis Mandel, Mark Jimenez, Emily Risley, Taishi Nammoto, Rebekka Williams, Max Panoff, Meynard Ballesteros, Timothy Kudryn, and Sebastian Carter.

FISHTRAC code incorporates HOTA evaluation scores from Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixe and Bastian Leibe, and their paper "HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking". That code from their repo is provided under the MIT License. We included a modified version of their code under the fishtrac/evaluation/TrackEval folder - any modifications we have made (to base it on DIOU, etc) are released under the same license.

Trackers:

All trackers are included under the fishtrac/trackers/<trackerName> folder unless otherwise specified.

Contact