CorrNet+
This repo holds codes of the paper: CorrNet+: Sign Language Recognition and Translation via Spatial-Temporal Correlation [paper], which is an extension of our previous work (CVPR 2023) [paper]
For the code supporting continuous sign language recognition, refer to CorrNet_Plus_CSLR for the code.
We currently reserve the code of CorrNet_Plus_SLT.
Performance
- On the continuous sign language cognition task, CorrNet+ achieves superior performance on PHOENIX14, PHOENIX14-T, CSL-Daily and CSL datasets.
| Method | PHOENIX2014 | PHOENIX2014-T | CSL-Daily | |||||
| Dev(%) | Test(%) | Dev(%) | Test(%) | Dev(%) | Test(%) | |||
| del/ins | WER | del/ins | WER | |||||
| CVT-SLR (CVPR2023) | 6.4/2.6 | 19.8 | 6.1/2.3 | 20.1 | 19.4 | 20.3 | - | - |
| CoSign-2s (ICCV2023) | - | 19.7 | - | 20.1 | 19.5 | 20.1 | - | - |
| AdaSize (PR2024) | 7.0/2.6 | 19.7 | 7.2/3.1 | 20.9 | 19.7 | 21.2 | 31.3 | 30.9 |
| AdaBrowse+ (ACMMM2023) | 6.0/2.5 | 19.6 | 5.9/2.6 | 20.7 | 19.5 | 20.6 | 31.2 | 30.7 |
| SEN (AAAI2023) | 5.8/2.6 | 19.5 | 7.3/4.0 | 21.0 | 19.3 | 20.7 | 31.1 | 30.7 |
| CTCA (CVPR2023) | 6.2/2.9 | 19.5 | 6.1/2.6 | 20.1 | 19.3 | 20.3 | 31.3 | 29.4 |
| C2SLR (CVPR2022) | - | 20.5 | - | 20.4 | 20.2 | 20.4 | - | - |
| CorrNet+ | 5.3/2.7 | 18.0 | 5.6/2.4 | 18.2 | 17.2 | 19.1 | 28.6 | 28.2 |
|---|---|---|---|---|---|---|---|---|
- On the sign language translation task, CorrNet+ achieves superior performance on PHOENIX14, PHOENIX14-T and CSL-Daily datasets.
| PHOENIX2014-T | ||||||||||
| Method | Dev(%) | Test(%) | ||||||||
| Rouge | BLEU1 | BLEU2 | BLEU3 | BLEU4 | Rouge | BLEU1 | BLEU2 | BLEU3 | BLEU4 | |
| SignBT (CVPR2021) | 50.29 | 51.11 | 37.90 | 29.80 | 24.45 | 49.54 | 50.80 | 37.75 | 29.72 | 24.32 |
| MMTLB (CVPR2022) | 53.10 | 53.95 | 41.12 | 33.14 | 27.61 | 52.65 | 53.97 | 41.75 | 33.84 | 28.39 |
| SLTUNET (ICLR2023) | 52.23 | - | - | - | 27.87 | 52.11 | 52.92 | 41.76 | 33.99 | 28.47 |
| TwoStream-SLT (NeuIPS2023) | 54.08 | 54.32 | 41.99 | 34.15 | 28.66 | 53.48 | 54.90 | 42.43 | 34.46 | 28.95 |
| CorrNet+ | 54.54 | 54.56 | 42.31 | 34.48 | 29.13 | 53.76 | 55.32 | 42.74 | 34.86 | 29.42 |
|---|---|---|---|---|---|---|---|---|---|---|
| CSL-Daily | ||||||||||
| Method | Dev(%) | Test(%) | ||||||||
| Rouge | BLEU1 | BLEU2 | BLEU3 | BLEU4 | Rouge | BLEU1 | BLEU2 | BLEU3 | BLEU4 | |
| SignBT (CVPR2021) | 49.49 | 51.46 | 37.23 | 27.51 | 20.80 | 49.31 | 51.42 | 37.26 | 27.76 | 21.34 |
| MMTLB (CVPR2022) | 53.38 | 53.81 | 40.84 | 31.29 | 24.42 | 53.25 | 53.31 | 40.41 | 30.87 | 23.92 |
| SLTUNET (ICLR2023) | 53.58 | - | - | - | 23.99 | 54.08 | 54.98 | 41.44 | 31.84 | 25.01 |
| TwoStream-SLT (NeuIPS2023) | 55.10 | 55.21 | 42.31 | 32.71 | 25.76 | 55.72 | 55.44 | 42.59 | 32.87 | 25.79 |
| CorrNet+ | 55.52 | 55.64 | 42.78 | 33.13 | 26.14 | 55.84 | 55.82 | 42.96 | 33.26 | 26.14 |
Visualizations
As shown below, our method intelligently models the human body trajectories across adjacent frames and pays special attention to the moving human body parts.
Data preparation, Environment, Training, Inference and Visualizations
For detailed instructions of data preparation, environment, training, inference and visualizations, please refer to each sub-repo for guidance.