add hota iou calculation along with detection/association accuracy

[Rusteam]

Closes #151

Following the author's blogpost to add HOTA calculation to this package.

Added hota_iou implementation only for a single selected iou threshold. Need some guidance on how to implement HOTA at range(0.05, 0.95, 0.05)

[cheind]

Hi, thanks a lot for your contribution!

[cheind]

I wonder if you could also add a HOTA metric to this

https://github.com/cheind/py-motmetrics/blob/8c25d76c03ad77e0c8a180717be75c53618a5541/motmetrics/tests/test_metrics.py#L422

test? I guess we could derive the necessary GT-values by running the files through the official implementation?

[Rusteam]

I wonder if you could also add a HOTA metric to this

https://github.com/cheind/py-motmetrics/blob/8c25d76c03ad77e0c8a180717be75c53618a5541/motmetrics/tests/test_metrics.py#L422

test? I guess we could derive the necessary GT-values by running the files through the official implementation?

should it be added to the default motchallenge metrics?

[cheind]

you mean to add it to mm.metrics.motchallenge_metrics?

I believe one could make a separate list of metrics for HOTA? such that we can write

summary = mh.compute_many(
        accs,
        metrics=mm.metrics.motchallenge_metrics + mm.metrics.hota_metrics,
        names=dnames,
        generate_overall=True,
    )

Alternatively, based on https://motchallenge.net/, it seems like MotChallenge is already reporting HOTA. So we could just add the metrics to the mm.metrics.motchallenge_metric list. What do you think?

[Rusteam]

you mean to add it to mm.metrics.motchallenge_metrics?

I believe one could make a separate list of metrics for HOTA? such that we can write

summary = mh.compute_many(
        accs,
        metrics=mm.metrics.motchallenge_metrics + mm.metrics.hota_metrics,
        names=dnames,
        generate_overall=True,
    )

Alternatively, based on https://motchallenge.net/, it seems like MotChallenge is already reporting HOTA. So we could just add the metrics to the mm.metrics.motchallenge_metric list. What do you think?

Since it's already included in the challenge, I can create hota_metrics with these 3 values and include it by default to motchallenge_metrics, however, it will be possible to use it standalone as well.

Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

[Rusteam]

These are results from the official repo on the same test files:

[cheind]

Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

[Rusteam]

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

I doubt because all of them are basically TP, FP and FN which are computed at a specified threshold.

I've noticed that my results of DetA and AssA are different from the orig. repo, need to look more into that.

[mikel-brostrom]

Really looking forward to start using py-motmetrics instead of trackeval for full MOTA, HOTA and IDF1 metrics! :rocket: This will make a huge impact in the accessibility and flexibility when it comes to this specific metric. Using trackeval is TEDIOUS.

[Rusteam]

I have followed the author's blog post in my implementation but the numbers are different from the original repository. Just need to find time to figure out where the catch is and how to compute a metric at multiple IOU thresholds in this package.

[mikel-brostrom]

If it is of any help, here you have the original HOTA implementation @Rusteam :

https://github.com/JonathonLuiten/TrackEval/blob/12c8791b303e0a0b50f753af204249e622d0281a/trackeval/metrics/hota.py#L9

[Rusteam]

Thanks. Yes I've seen it but it requires time to find the difference

[Justin900429]

Hi,

I have tried combining different $\alpha$ by:

import os

import numpy as np

import motmetrics as mm

"""Tests metrics for sequences TUD-Campus and TUD-Stadtmitte."""
dnames = [
    "TUD-Stadtmitte",
]

DATA_DIR = "motmetrics/data"

def compute_motchallenge(dname):
    df_gt = mm.io.loadtxt(os.path.join(dname, "gt.txt"))
    df_test = mm.io.loadtxt(os.path.join(dname, "test.txt"))

    res_file = []

    for th in np.arange(0.05, 0.99, 0.05):
        res_file.append(mm.utils.compare_to_groundtruth(df_gt, df_test, "iou", distth=th))
    return res_file

accs = [compute_motchallenge(os.path.join(DATA_DIR, d)) for d in dnames]
det = []
mh = mm.metrics.create()

for idx in range(len(accs[0])):
    summary = mh.compute_many(
        [accs[0][idx]],
        metrics=["det_acc"],
        names=[dnames[0]],
        generate_overall=False,
    )
    det.append(float(summary["det_acc"].iloc[0]))

print(sum(det) / len(det))

and found the DeT is different: 39.227 (official) v.s. 39.87 (this) in TUD-Stadtmitte. I guess this different is caused by the reweighting before filtering and association implemented in the official repo. Check this line.

---

Also I need some hint how to implement the actual HOTA which is an average of hota_iou across a range of iou thresholds.

I hopefully can have a look at that in the next week. Do you know if its possible pre-compute some values about Det(alpha) Ass(alpha), so that we don't have to run the entire computation for every alpha?

Computing the iou_matrix (of dist) once and use different $alpha$ for filtering might speed up just like the official does. This require to re-write some of the functions (such as mm.utils.compare_to_groundtruth) to support this kind of operation.

---

Based on these reasons, maybe we can do the HOTA computation with a indepdent function without changing the current codebase?

[cheind]

@Rusteam , @Justin900429 how does your PR relate to this one?

[Rusteam]

He is right. There's some difference. I wonder how to fix it

[johnnynunez]

any news?