A General Framework for Jersey Number Recognition in Sports

Code, data, and model weights for paper A General Framework for Jersey Number Recognition in Sports (Maria Koshkina, James H. Elder).

Pipeline

Image-level detection, localization and recognition (experiments on Hockey dataset):

legibility classifier
scene text recognition for jersey numbers

Tracklet-level detection, localization and recognition (experiments on SoccerNet dataset):

occlusion/outlier removal using re-id features and fitting a Gaussian
legibility classifier
pose-guided RoI cropping
scene text recognition for jersey numbers
tracklet prediction consolidation

Requirements:

pytorch 1.9.0
opencv

Setup:

Clone current repo. Create conda environment and install requirements. Code makes use of the several repositories. Run

python3 setup.py

to automatically clone, setup a separate conda environment for each and fetch models.

Alternatively, clone each of the following repo, setup conda environments for each following documentation in corresponding repo, and download models:

SAM:

Should be in jersey-number-pipeline/sam. Repo: https://github.com/davda54/sam

Centroid-Reid:

Should be in jersey-number-pipeline/reid/centroids-reid. Repo: https://github.com/mikwieczorek/centroids-reid. Download centroid-reid model weights and place them under jersey-number-pipeline/reid/centroids-reid/models.

ViTPose:

Should be in jersey-number-pipeline/pose/ViTPose. Repo: https://github.com/ViTAE-Transformer/ViTPose. Download ViTPose model weights and place them under jersey-number-pipeline/pose/ViTPose/checkpoints/.

PARSeq:

Should be in jersey-number-pipeline/str/parseq. Repo: https://github.com/baudm/parseq. Model weights should be downloaded and placed under jersey-number-pipeline/models/.

Data:

SoccerNet Jersey Number Recognition: https://github.com/SoccerNet/sn-jersey Download and save under /data subfolder.

Weakly-labelled player images used to train legibility classifier can be downloaded here.
Weakly-labelled jersey number crops used to fine-tune STR in LMDB format can be downloaded here.

Hockey (comprised of legibility dataset and jersey number dataset):

Request access by contacting Maria Koshkina. Extract under data/Hockey subfolder.

Trained Legibility Classifier Weights:

Download and place under jersey-number-pipeline/models/.

Configuration:

Update configuration.py if required to set custom path to data or dependencies.

Inference:

To run the full inference pipeline for SoccerNet:

python3 main.py SoccerNet test

To run legibility and jersey number inference for hockey:

python3 main.py Hockey test

Update actions in main.py actions list to run steps selectively.

Train (Hockey)

Train legibility classifier:

python3 legibility_classifier.py --train --arch resnet34 --sam --data <new-dataset-directory> --trained_model_path ./experiments/hockey_legibility.pth

Fine-tune PARSeq STR for hockey number recognition:

python3 main.py Hockey train --train_str

Trained model will be under str/parseq/outputs

Train (SoccerNet)

To train legibility classifier and jersey number recognition for SoccerNet, we first generate weakly labelled datasets and then use them to fine-tune. Weak labels are obtained by using models trained on hockey data.

Train legibility classifier for it:

python3 legibility_classifier.py --finetune --arch resnet34 --sam --data <new-dataset-directory>  --full_val_dir
<new-dataset-directory>/val --trained_model_path ./experiments/hockey_legibility.pth --new_trained_model_path ./experiments/sn_legibility.pth

Fine-tune PARSeq on weakly-labelled SoccerNet data:

python3 main.py SoccerNet train --train_str

Trained model will be under str/parseq/outputs.

Citation

@InProceedings{Koshkina_2024_CVPR,
    author    = {Koshkina, Maria and Elder, James H.},
    title     = {A General Framework for Jersey Number Recognition in Sports Video},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2024},
    pages     = {3235-3244}
}

Acknowledgements

We would like to thank authors of the following repositories:

License

This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.

mkoshkina / jersey-number-pipeline

readme