search

mikel-brostrom / boxmot

BoxMOT: pluggable SOTA tracking modules for segmentation, object detection and pose estimation models
GNU Affero General Public License v3.0
6.65k stars 1.7k forks source link

Evaluation on custom data #921

Closed sivaji123256 closed 1 year ago

sivaji123256 commented 1 year ago

Search before asking

Question

Hi @gkeechin @Borda @hdnh2006 , While I was trying to run the val.py both on mot17-mini and my own custom data ,I was facing the following issue. How can I fix this ? Traceback (most recent call last): File "/home/ubuntu/yolo_tracking/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 42, in import trackeval # noqa: E402 ModuleNotFoundError: No module named 'trackeval'

Traceback (most recent call last): File "/home/ubuntu/yolo_tracking/yolo_tracking/examples/val.py", line 345, in e.run(opt) File "/home/ubuntu/yolo_tracking/yolo_tracking/examples/val.py", line 302, in run writer.add_scalar('HOTA', combined_results['HOTA']) KeyError: 'HOTA'

I have cloned the trackeval repo into val_utils folder and ran pip install requirements.txt as well.But still ,I was facing the same issue. Thanks in advance .

mikel-brostrom commented 1 year ago

The val.py script is now less confusing. The execution stops whenever there is ANY error and prompts it clearly. This was not the case previously. When a single tracking thread could crash but the metrics got calculated anyways. Please git pull and run again. Post your error.

sivaji123256 commented 1 year ago

Hi @mikel-brostrom , I ran it and the error is as follows: val: yolo_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/yolov8l_sheep_dog_human_13042023.pt, reid_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/osnet_x1_0_imagenet.pt, tracking_method=ocsort, name=exp, project=/home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=MOTCUSTOM, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2 2023-06-05 13:36:13.665 | INFO | main:download_mot_eval_tools:64 - Eval repo already downloaded 2023-06-05 13:36:13.666 | INFO | main:eval:195 - Staring evaluation process on /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-06/MOTCUSTOM-06 2023-06-05 13:41:25.615 | SUCCESS | main:eval:227 - /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-06/MOTCUSTOM-06 evaluation succeeded 2023-06-05 13:41:25.615 | INFO | main:eval:195 - Staring evaluation process on /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-05/MOTCUSTOM-05 2023-06-05 13:46:47.798 | SUCCESS | main:eval:227 - /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-05/MOTCUSTOM-05 evaluation succeeded val: yolo_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/yolov8l_sheep_dog_human_13042023.pt, reid_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/osnet_x1_0_imagenet.pt, tracking_method=ocsort, name=exp, project=/home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=MOTCUSTOM, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2 2023-06-05 13:46:48.693 | ERROR | main:eval:260 - multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap return f(args, kw) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/eval.py", line 222, in eval_sequence data = dataset.get_preprocessed_seq_data(raw_data, cls) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap return f(*args, **kw) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 366, in get_preprocessed_seq_data raise(TrackEvalException('Attempting to evaluate using invalid gt classes. ' trackeval.utils.TrackEvalException: Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 831: -1 """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/eval.py", line 102, in evaluate results = pool.map(_eval_sequence, seq_list) File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 771, in get raise self._value trackeval.utils.TrackEvalException: Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 831: -1 multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(args)) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap return f(args, kw) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/eval.py", line 222, in eval_sequence data = dataset.get_preprocessed_seq_data(raw_data, cls) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap return f(*args, **kw) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py", line 366, in get_preprocessed_seq_data raise(TrackEvalException('Attempting to evaluate using invalid gt classes. ' trackeval.utils.TrackEvalException: Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 831: -1 """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/scripts/run_mot_challenge.py", line 91, in evaluator.evaluate(dataset_list, metrics_list) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/_timing.py", line 64, in wrap return f(*args, **kw) File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/eval.py", line 207, in evaluate raise err File "/home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/trackeval/eval.py", line 102, in evaluate results = pool.map(_eval_sequence, seq_list) File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/pool.py", line 771, in get raise self._value trackeval.utils.TrackEvalException: Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 831: -1

2023-06-05 13:46:48.693 | ERROR | main:eval:261 - Eval Config: USE_PARALLEL : True
NUM_PARALLEL_CORES : 4
BREAK_ON_ERROR : True
RETURN_ON_ERROR : False
LOG_ON_ERROR : /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/error_log.txt PRINT_RESULTS : True
PRINT_ONLY_COMBINED : False
PRINT_CONFIG : True
TIME_PROGRESS : True
DISPLAY_LESS_PROGRESS : False
OUTPUT_SUMMARY : True
OUTPUT_EMPTY_CLASSES : True
OUTPUT_DETAILED : True
PLOT_CURVES : True

MotChallenge2DBox Config: PRINT_CONFIG : True
GT_FOLDER : /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test TRACKERS_FOLDER : /home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val/exp3 OUTPUT_FOLDER : None
TRACKERS_TO_EVAL : ['labels']
CLASSES_TO_EVAL : ['pedestrian']
BENCHMARK :
SPLIT_TO_EVAL : train
INPUT_AS_ZIP : False
DO_PREPROC : True
TRACKER_SUB_FOLDER :
OUTPUT_SUB_FOLDER :
TRACKER_DISPLAY_NAMES : None
SEQMAP_FOLDER : None
SEQMAP_FILE : None
SEQ_INFO : {'MOTCUSTOM-06': None, 'MOTCUSTOM-05': None} GT_LOC_FORMAT : {gt_folder}/{seq}/gt/gt.txt
SKIP_SPLIT_FOL : True

CLEAR Config: METRICS : ['HOTA', 'CLEAR', 'Identity'] THRESHOLD : 0.5
PRINT_CONFIG : True

Identity Config: METRICS : ['HOTA', 'CLEAR', 'Identity'] THRESHOLD : 0.5
PRINT_CONFIG : True

Evaluating 1 tracker(s) on 2 sequence(s) for 1 class(es) on MotChallenge2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count

Evaluating labels

-1 Tracker labels was unable to be evaluated. Attempting to evaluate using invalid gt classes. This warning only triggers if preprocessing is performed, e.g. not for MOT15 or where prepropressing is explicitly disabled. Please either check your gt data, or disable preprocessing. The following invalid classes were found in timestep 831: -1

How to disable preprocessing.I have gone through the arguments of evolve.py and val.py.But couldn't find any arguments corresponding to it.

sivaji123256 commented 1 year ago

@mikel-brostrom , I made the do_prepoc = False in eval.py in the following file /yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py I was able to run it successfully. val: yolo_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/yolov8l_sheep_dog_human_13042023.pt, reid_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/osnet_x1_0_imagenet.pt, tracking_method=ocsort, name=exp, project=/home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=MOTCUSTOM, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2 2023-06-05 14:21:54.352 | INFO | main:download_mot_eval_tools:64 - Eval repo already downloaded 2023-06-05 14:21:54.353 | INFO | main:eval:195 - Staring evaluation process on /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-06/MOTCUSTOM-06 2023-06-05 14:27:07.013 | SUCCESS | main:eval:227 - /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-06/MOTCUSTOM-06 evaluation succeeded 2023-06-05 14:27:07.013 | INFO | main:eval:195 - Staring evaluation process on /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-05/MOTCUSTOM-05 2023-06-05 14:32:29.867 | SUCCESS | main:eval:227 - /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test/MOTCUSTOM-05/MOTCUSTOM-05 evaluation succeeded val: yolo_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/yolov8l_sheep_dog_human_13042023.pt, reid_model=/home/ubuntu/yolo_tracking/yolo_tracking/examples/weights/osnet_x1_0_imagenet.pt, tracking_method=ocsort, name=exp, project=/home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val, exist_ok=False, benchmark=MOTCUSTOM, split=test, eval_existing=False, conf=0.45, imgsz=[1280], device=[''], processes_per_device=2 2023-06-05 14:32:32.935 | INFO | main:eval:264 - Eval Config: USE_PARALLEL : True
NUM_PARALLEL_CORES : 4
BREAK_ON_ERROR : True
RETURN_ON_ERROR : False
LOG_ON_ERROR : /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/error_log.txt PRINT_RESULTS : True
PRINT_ONLY_COMBINED : False
PRINT_CONFIG : True
TIME_PROGRESS : True
DISPLAY_LESS_PROGRESS : False
OUTPUT_SUMMARY : True
OUTPUT_EMPTY_CLASSES : True
OUTPUT_DETAILED : True
PLOT_CURVES : True

MotChallenge2DBox Config: PRINT_CONFIG : True
GT_FOLDER : /home/ubuntu/tracking_tuning/yolo_tracking/examples/val_utils/data/MOTCUSTOM/test TRACKERS_FOLDER : /home/ubuntu/tracking_tuning/yolo_tracking/examples/runs/val/exp5 OUTPUT_FOLDER : None
TRACKERS_TO_EVAL : ['labels']
CLASSES_TO_EVAL : ['pedestrian']
BENCHMARK :
SPLIT_TO_EVAL : train
INPUT_AS_ZIP : False
DO_PREPROC : True
TRACKER_SUB_FOLDER :
OUTPUT_SUB_FOLDER :
TRACKER_DISPLAY_NAMES : None
SEQMAP_FOLDER : None
SEQMAP_FILE : None
SEQ_INFO : {'MOTCUSTOM-06': None, 'MOTCUSTOM-05': None} GT_LOC_FORMAT : {gt_folder}/{seq}/gt/gt.txt
SKIP_SPLIT_FOL : True

CLEAR Config: METRICS : ['HOTA', 'CLEAR', 'Identity'] THRESHOLD : 0.5
PRINT_CONFIG : True

Identity Config: METRICS : ['HOTA', 'CLEAR', 'Identity'] THRESHOLD : 0.5
PRINT_CONFIG : True

Evaluating 1 tracker(s) on 2 sequence(s) for 1 class(es) on MotChallenge2DBox dataset using the following metrics: HOTA, CLEAR, Identity, Count

Evaluating labels

All sequences for labels finished in 1.82 seconds

HOTA: labels-pedestrian HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0) MOTCUSTOM-05 48.978 35.101 68.35 36.061 87.731 79.056 75.41 88.761 49.642 56.339 86.218 48.575
MOTCUSTOM-06 28.315 18.885 42.499 19.695 78.17 54.159 62.397 88.411 28.924 32.449 81.292 26.378
COMBINED 38.742 25.981 57.788 26.92 83.555 68.881 70.095 88.613 39.439 44.529 84.133 37.463

CLEAR: labels-pedestrian MOTA MOTP MODA CLR_Re CLR_Pr MTR PTR MLR sMOTA CLR_TP CLR_FN CLR_FP IDSW MT PT ML Frag
MOTCUSTOM-05 37.595 87.706 39.473 40.288 98.015 46.809 12.234 40.957 32.642 3605 5343 73 168 176 46 154 217
MOTCUSTOM-06 17.102 87.718 18.746 21.97 87.202 10.624 39.723 49.654 14.404 2487 8833 365 186 46 172 215 280
COMBINED 26.15 87.711 27.896 30.057 93.292 27.441 26.947 45.612 22.456 6092 14176 438 354 222 218 369 497

Identity: labels-pedestrian IDF1 IDR IDP IDTP IDFN IDFP
MOTCUSTOM-05 49.501 34.924 84.965 3125 5823 553
MOTCUSTOM-06 25.882 16.201 64.306 1834 9486 1018
COMBINED 37.01 24.467 75.942 4959 15309 1571

Count: labels-pedestrian Dets GT_Dets IDs GT_IDs
MOTCUSTOM-05 3678 8948 147 376
MOTCUSTOM-06 2852 11320 214 433
COMBINED 6530 20268 361 809

classes_to_eval = [pedestrian] , but my class name was different.I couldn't analyze the results completely as the classes_to_eval having its default value.Can you help me in analyzing the above result?

mikel-brostrom commented 1 year ago

I made the do_prepoc = False in eval.py in the following file /yolo_tracking/examples/val_utils/trackeval/datasets/mot_challenge_2d_box.py I was able to run it successfully.

Thanks! I don't get much feedback on custom tracking datasets so it is hard to catch all the corner cases in order to generalize the script.

MOTA and IDF1 overemphasize detection and association respectively. HOTA explicitly measures both types of errors and combines these in a balanced way. HOTA also incorporates measuring the localization accuracy of tracking results which isn’t present in either MOTA or IDF1.

Overall, your performance looks worse for your second sequence.

ImSuMyatNoe commented 1 year ago

Hi @sivaji123256 I would like to ask a question that how do you tain on the custom data? I do not find the train code (python file) to train with the yolov8x-seg.pt for my custom data? It would be great if you would answer this?

mikel-brostrom commented 1 year ago

Custom evaluation works much better here. Not merged as HOTA is not calculatable with the motmetrics pip package