MEMOTR evaluation on MOT17

[sparshgarg23]

I am trying to run the evaluation code on MOT17 using my own checkpoint for the MEMOTR model,however every time I try running the evaluation code using the below arguments,the configuration file that is loaded ends up pointing to the Dancetrack dataset. Could you tell me how to change the arguments or config file so that the evaluation can be done on MOT17.

I saw there is a specific command in eval_engine that runs the evaluation on MOT17,and worst case scenario I am thinking of directly running that command.

This is the command for evaluation

!python main.py --mode eval --data-root /content/MeMOTR/DATADIR/ --eval-mode specific --eval-model /content/MeMOTR/outputs/memotr_mot17/checkpoint_59.pth  --eval-dir /content/MeMOTR/outputs/memotr_mot17/train --eval-threads 0

and this is the error log i get

2024-06-07 05:18:43.709544: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-06-07 05:18:43.763543: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-06-07 05:18:43.763595: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-06-07 05:18:43.765557: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-07 05:18:43.774258: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-07 05:18:44.912419: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
===>  Running checkpoint '/content/MeMOTR/outputs/memotr_mot17/checkpoint_59.pth'
2024-06-07 05:18:49.724680: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-06-07 05:18:49.724733: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-06-07 05:18:49.725913: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-06-07 05:18:50.837143: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Configs: {'GIT_VERSION': None, 'MODE': 'submit', 'CONFIG_PATH': './configs/train_dancetrack.yaml', 'VISUALIZE': False, 'AVAILABLE_GPUS': '0,1,2,3,4,5,6,7', 'DEVICE': 'cuda', 'OUTPUTS_DIR': './outputs/pycharm/', 'USE_DISTRIBUTED': False, 'USE_CHECKPOINT': False, 'CHECKPOINT_LEVEL': 2, 'RESUME': None, 'RESUME_SCHEDULER': True, 'MULTI_CHECKPOINT': False, 'SUBMIT_DIR': '/content/MeMOTR/outputs/memotr_mot17/train', 'SUBMIT_MODEL': '/content/MeMOTR/outputs/memotr_mot17/checkpoint_59.pth', 'SUBMIT_DATA_SPLIT': 'val', 'DET_SCORE_THRESH': 0.5, 'TRACK_SCORE_THRESH': 0.5, 'RESULT_SCORE_THRESH': 0.5, 'MISS_TOLERANCE': 30, 'USE_MOTION': False, 'MOTION_MIN_LENGTH': 3, 'MOTION_MAX_LENGTH': 5, 'MOTION_LAMBDA': 0.5, 'EVAL_DIR': None, 'EVAL_MODE': 'specific', 'EVAL_MODEL': None, 'EVAL_PORT': None, 'EVAL_THREADS': 1, 'EVAL_DATA_SPLIT': 'val', 'DATASET': 'DanceTrack', 'USE_MOTSYNTH': None, 'USE_CROWDHUMAN': None, 'MOTSYNTH_RATE': None, 'DATA_ROOT': '/content/MeMOTR/DATADIR/', 'DATA_PATH': None, 'NUM_WORKERS': 4, 'BATCH_SIZE': 1, 'ACCUMULATION_STEPS': 1, 'COCO_SIZE': False, 'OVERFLOW_BBOX': False, 'REVERSE_CLIP': 0.0, 'BACKBONE': 'resnet50', 'HIDDEN_DIM': 256, 'FFN_DIM': 2048, 'NUM_FEATURE_LEVELS': 4, 'NUM_HEADS': 8, 'NUM_ENC_POINTS': 4, 'NUM_DEC_POINTS': 4, 'NUM_ENC_LAYERS': 6, 'NUM_DEC_LAYERS': 6, 'MERGE_DET_TRACK_LAYER': 1, 'ACTIVATION': 'ReLU', 'RETURN_INTER_DEC': True, 'EXTRA_TRACK_ATTN': False, 'AUX_LOSS': True, 'USE_DAB': True, 'UPDATE_THRESH': 0.5, 'LONG_MEMORY_LAMBDA': 0.01, 'PRETRAINED_MODEL': 'dab_deformable_detr.pth', 'SAMPLE_STEPS': [6, 10, 14], 'SAMPLE_LENGTHS': [2, 3, 4, 5], 'SAMPLE_MODES': ['random_interval'], 'SAMPLE_INTERVALS': [10], 'SEED': 42, 'EPOCHS': 20, 'ONLY_TRAIN_QUERY_UPDATER_AFTER': 20, 'DROPOUT': 0.0, 'NUM_DET_QUERIES': 300, 'TP_DROP_RATE': 0.0, 'FP_INSERT_RATE': 0.0, 'LR': 0.0002, 'LR_BACKBONE': 2e-05, 'LR_POINTS': 1e-05, 'WEIGHT_DECAY': 0.0005, 'CLIP_MAX_NORM': 0.1, 'LR_SCHEDULER': 'MultiStep', 'LR_DROP_RATE': 0.1, 'LR_DROP_MILESTONES': [12], 'MATCH_COST_CLASS': 2, 'MATCH_COST_BBOX': 5, 'MATCH_COST_GIOU': 2, 'LOSS_WEIGHT_FOCAL': 2, 'LOSS_WEIGHT_L1': 5, 'LOSS_WEIGHT_GIOU': 2, 'AUX_LOSS_WEIGHT': [1.0, 1.0, 1.0, 1.0, 1.0]}
Traceback (most recent call last):
  File "/content/MeMOTR/main.py", line 124, in <module>
    main(config=merged_config)
  File "/content/MeMOTR/main.py", line 109, in main
    submit(config=config)
  File "/content/MeMOTR/submit_engine.py", line 195, in submit
    train_config = yaml_to_dict(path=path.join(config["SUBMIT_DIR"], "train/config.yaml"))
  File "/content/MeMOTR/utils/utils.py", line 53, in yaml_to_dict
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/content/MeMOTR/outputs/memotr_mot17/train/train/config.yaml'
mv: cannot stat '/content/MeMOTR/outputs/memotr_mot17/train/val/tracker': No such file or directory
/content/MeMOTR/TrackEval/trackeval/eval.py:99: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  seq_key is not 'COMBINED_SEQ'}
/content/MeMOTR/TrackEval/trackeval/utils.py:137: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if (len(current_values) == len(keys)) and seq is not '':
/content/MeMOTR/TrackEval/trackeval/datasets/mot_challenge_2d_box.py:151: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if self.config["SEQMAP_FOLDER"] is 'None':

Eval Config:
USE_PARALLEL         : True                          
NUM_PARALLEL_CORES   : 8                             
BREAK_ON_ERROR       : True                          
RETURN_ON_ERROR      : False                         
LOG_ON_ERROR         : /content/MeMOTR/TrackEval/error_log.txt
PRINT_RESULTS        : True                          
PRINT_ONLY_COMBINED  : False                         
PRINT_CONFIG         : True                          
TIME_PROGRESS        : True                          
DISPLAY_LESS_PROGRESS : False                         
OUTPUT_SUMMARY       : True                          
OUTPUT_EMPTY_CLASSES : True                          
OUTPUT_DETAILED      : True                          
PLOT_CURVES          : False                         

MotChallenge2DBox Config:
PRINT_CONFIG         : True                          
GT_FOLDER            : /content/MeMOTR/DATADIR/DanceTrack/val
TRACKERS_FOLDER      : /content/MeMOTR/outputs/memotr_mot17/checkpoint_59_tracker
OUTPUT_FOLDER        : None                          
TRACKERS_TO_EVAL     : ['']                          
CLASSES_TO_EVAL      : ['pedestrian']                
BENCHMARK            : MOT17                         
SPLIT_TO_EVAL        : val                           
INPUT_AS_ZIP         : False                         
DO_PREPROC           : True                          
TRACKER_SUB_FOLDER   :                               
OUTPUT_SUB_FOLDER    :                               
TRACKER_DISPLAY_NAMES : None                          
SEQMAP_FOLDER        : None                          
SEQMAP_FILE          : /content/MeMOTR/DATADIR/DanceTrack/val_seqmap.txt
SEQ_INFO             : None                          
GT_LOC_FORMAT        : {gt_folder}/{seq}/gt/gt.txt   
SKIP_SPLIT_FOL       : True                          
no seqmap found: /content/MeMOTR/DATADIR/DanceTrack/val_seqmap.txt
Traceback (most recent call last):
  File "/content/MeMOTR/TrackEval/scripts/run_mot_challenge.py", line 84, in <module>
    dataset_list = [trackeval.datasets.MotChallenge2DBox(dataset_config)]
  File "/content/MeMOTR/TrackEval/trackeval/datasets/mot_challenge_2d_box.py", line 82, in __init__
    self.seq_list, self.seq_lengths = self._get_seq_info()
  File "/content/MeMOTR/TrackEval/trackeval/datasets/mot_challenge_2d_box.py", line 157, in _get_seq_info
    raise TrackEvalException('no seqmap found: ' + os.path.basename(seqmap_file))
trackeval.utils.TrackEvalException: no seqmap found: val_seqmap.txt
Traceback (most recent call last):
  File "/content/MeMOTR/main.py", line 124, in <module>
    main(config=merged_config)
  File "/content/MeMOTR/main.py", line 111, in main
    evaluate(config=config)
  File "/content/MeMOTR/eval_engine.py", line 36, in evaluate
    metrics = eval_model(model=config["EVAL_MODEL"], eval_dir=eval_dir,
  File "/content/MeMOTR/eval_engine.py", line 118, in eval_model
    with open(metric_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/content/MeMOTR/outputs/memotr_mot17/checkpoint_59_tracker/pedestrian_summary.txt'

not sure at which point the dancetrack configuraiton was referred to here,because I am using the MOT17 dataset.

[HELLORPG]

For inference on MOT17, you should add a runtime option --config-path <your training config file path> to the submitting and evaluation scripts. Sorry for the insufficient details. I will make this clearer in our documents later.