VlSomers
commented
3 months ago Hi @frj555, this is a strange issue that is difficult to analyse without further information. It looks like the GS-HOTA is 0 because the pipeline did not produce any detection. Can you please answer the following questions:
- Are Sn-Gamestate, Tracklab, and all other dependencies up to date? Especially Trackeval (https://github.com/SoccerNet/sn-trackeval). Please make sure to have everything up to date. Can you copy paste the versions of libraries in your current conda/poetry/pip environment here?
- Did you change anything in the codebase or have you just run the latest clone of the repository without modifications?
- Can you have a look at the finale produced video? If I'm not wrong, you should see no detections, but maybe pitch lines?
frj555
victorjoos
(tracklab) PS X:\Pycharmproj\Sngamestate\sn-gamestate> python -m tracklab.main -cn soccernet [2024-05-27 17:44:38,788][main][INFO] - Using device: 'cuda'. Loading SoccerNetGS 'train' set videos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:06 Loading SoccerNetGS 'valid' set videos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:06 Loading SoccerNetGS 'test' set videos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:06 Loading SoccerNetGS 'challenge' set videos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:06 Overwriting current config with config loaded from X://Pycharmproj/Sngamestate/sn-gamestate/pretrained_models/reid/prtreid-soccernet-baseline.pth.tar Diff from default config : {'batch_size': 32, 'ce': 0.0, 'dim_reduce_output': 256, 'hrnet_pretrained_path': 'X://Pycharmproj/Sngamestate/sn-gamestate/pretrained_models/reid', 'id': 0.0, 'load_config': True, 'mask_filtering_testing': False, 'max_epoch': 20, 'preprocess': 'id', 'test_embeddings': "['globl']", 'tr': 0.0, 'train_sampler': 'PrtreidSampler', 'train_sampler_t': 'PrtreidSampler'} Loads checkpoint by http backend from path: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth The model and loaded state dict do not match exactly
unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std
05/27 17:45:15 - mmengine - WARNING - Failed to search registry with scope "mmocr" in the "function" registry tree. As a workaround, the current "function" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmocr" is a correct scope, or whether the registry is initialized. Loads checkpoint by http backend from path: https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real_20220915_171910-04eb4e75.pth The model and loaded state dict do not match exactly
unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmocr/textdet/dbnet/dbnet_resnet18_fpnc_1200e_icdar2015/dbnet_resnet18_fpnc_1200e_icdar2015_20220825_221614-7c0e94f2.pth The model and loaded state dict do not match exactly
unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmocr/textrecog/sar/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real/sar_resnet31_parallel-decoder_5e_st-sub_mj-sub_sa_real_20220915_171910-04eb4e75.pth The model and loaded state dict do not match exactly
unexpected key in source state_dict: data_preprocessor.mean, data_preprocessor.std
[05/27/24 17:45:17] INFO Pipeline: YOLOv8 -> PRTReId -> BPBReIDStrongSORT -> TVCalib_Segmentation -> TVCalib -> MMOCR -> MajorityVoteTracklet -> TrackletTeamClustering -> TrackletTeamSideLabeling module.py:68 INFO Starting tracking operation on valid set. main.py:47 INFO Saving TrackerState to X:\Pycharmproj\Sngamestate\sn-gamestate\outputs\sn-gamestate\2024-05-27\17-44-38\states\sn-gamestate.pklz tracker_state.py:45 INFO Pipeline has been validated module.py:85 building model on device cuda => init weights from normal distribution Loading pretrained ImageNet HRNet32 model at X://Pycharmproj/Sngamestate/sn-gamestate/pretrained_models/reid\hrnetv2_w32_imagenet_pretrained.pth => loading pretrained model X://Pycharmproj/Sngamestate/sn-gamestate/pretrained_models/reid\hrnetv2_w32_imagenet_pretrained.pth Successfully loaded pretrained weights from "X://Pycharmproj/Sngamestate/sn-gamestate/pretrained_models/reid/prtreid-soccernet-baseline.pth.tar" ** The following layers are discarded due to unmatched keys or layer size: ['global_identity_classifier.classifier.weight', 'background_identity_classifier.classifier.weight', 'foreground_identity_classifier.classifier.weight', 'concat_parts_identity_classifier.classifier.weight', 'parts_identity_classifier.0.classifier.weight'] Building train transforms ...
Building test transforms ...
Tracking videos (SNGS-021) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1/1 0:52:08 • 0:00:00 [05/27/24 18:40:13] INFO Starting evaluation. main.py:107 INFO Starting evaluation using TrackEval library (https://github.com/JonathonLuiten/TrackEval) trackeval_evaluator.py:28 INFO Tracking predictions saved in SoccerNetGS format in eval\pred\SoccerNetGS-valid\tracklab trackeval_evaluator.py:45 INFO Tracking ground truth saved in SoccerNetGS format in eval\pred\SoccerNetGS-valid\tracklab trackeval_evaluator.py:65 Initializing the dataset class for the SoccerNet Game State Reconstruction task. IMPORTANT: The official evaluation metric for the task, i.e. the 'GS-HOTA' will appear under the 'HOTA' name in the evaluation script output. This happen because GS-HOTA mainly uses the same logic as the HOTA metric, the HOTA evaluation class is therefore not forked but re-used. The key practical difference between the GS-HOTA and the HOTA is actually the similarity metric used to match predictions with ground truth.Since this similarity score is computed outside the HOTA class (i.e. inside the SoccerNetGS dataset class), there was no need to fork it into a GS-HOTA class. Please refer to the official paper for more information. Using a sigma of 2.042694913268175 for the gaussian similarity metric, based on a distance tolerance of 5 meters.
Evaluating 1 tracker(s) on 1 sequence(s) for 1 class(es) on SoccerNetGS dataset using the following metrics: HOTA, Identity, Count
Evaluating tracklab
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 14.70it/s]
HOTA: tracklab-cls_comb_cls_av HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0) COMBINED 0 0 0 0 0 0 0 100 0 0 100 0
Identity: tracklab-cls_comb_cls_av IDF1 IDR IDP IDTP IDFN IDFP COMBINED 0 0 0 0 2067 0
Count: tracklab-cls_comb_cls_av Dets GT_Dets IDs GT_IDs COMBINED 0 2067 0 19
HOTA: tracklab-cls_comb_det_av HOTA DetA AssA DetRe DetPr AssRe AssPr LocA OWTA HOTA(0) LocA(0) HOTALocA(0) COMBINED 0 0 0 0 0 0 0 100 0 0 100 0
Identity: tracklab-cls_comb_det_av IDF1 IDR IDP IDTP IDFN IDFP COMBINED 0 0 0 0 2067 0
Count: tracklab-cls_comb_det_av Dets GT_Dets IDs GT_IDs COMBINED 0 2067 0 19 [05/27/24 18:40:15] INFO SoccerNet Game State Reconstruction performance GS-HOTA = 0% (config: EVAL_SPACE=pitch, USE_JERSEY_NUMBERS=True, USE_TEAMS=True, USE_ROLES=True, EVAL_DIST_TOL=5) soccernet_game_state.py:48 INFO Have a look at 'tracklab/tracklab/configs/dataset/soccernet_gs.yaml' for more details about the GS-HOTA metric and the evaluation configuration. soccernet_game_state.py:49 INFO Saved state at : X:\Pycharmproj\Sngamestate\sn-gamestate\outputs\sn-gamestate\2024-05-27\17-44-38\states\sn-gamestate.pklz main.py:66 [W C:\cb\pytorch_1000000000000\work\torch\csrc\CudaIPCTypes.cpp:95] Producer process tried to deallocate over 1000 memory blocks referred by consumer processes. Deallocation might be significantly slowed down. We assume it will never going to be the case, but if it is, please file but to https://github.com/pytorch/pytorch [W C:\cb\pytorch_1000000000000\work\torch\csrc\CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors] [W CUDAGuardImpl.h:46] Warning: CUDA warning: driver shutting down (function uncheckedGetDevice) ....