Overview

Introduction

This is the official implementation of the AAAI 22 accepted paper : Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition. paper

Abstract

Existing Scene Text Recognition (STR) methods typically use a language model to optimize the joint probability of the 1D character sequence predicted by a visual recognition (VR) model, which ignore the 2D spatial context of visual semantics within and between character instances, making them not generalize well to arbitrary shape scene text. To address this issue, we make the first attempt to perform textual reasoning based on visual semantics in this paper. Technically, given the character segmentation maps predicted by a VR model, we construct a subgraph for each instance, where nodes represent the pixels in it and edges are added between nodes based on their spatial similarity. Then, these subgraphs are sequentially connected by their root nodes and merged into a complete graph. Based on this graph, we devise a graph convolutional network for textual reasoning (GTR) by supervising it with a cross-entropy loss. GTR can be easily plugged in representative STR models to improve their performance owing to better textual reasoning. Specifically, we construct our model, namely S-GTR, by paralleling GTR to the language model in a segmentation-based STR baseline, which can effectively exploit the visual-linguistic complementarity via mutual learning. S-GTR sets new state-of-the-art on six challenging STR benchmarks and generalizes well to multi-linguistic datasets.

Framework

How to use

Env

PyTorch == 1.1.0 
torchvision == 0.3.0
fasttext == 0.9.1

Details can be found in requirements.txt

Train

Prepare your data

• Download the training set from Synthesis training dataset: Baidu(key:c83d) and Real training datasetBaidu(key:datm)
• Download the pretrained Seg-baseline visual recognition model from here(soon update)
• Update the path in the lib/tools/create_all_synth_lmdb.py
• Run the lib/tools/create_all_synth_lmdb.py
• Note: it may result in large storage space, you can modify the datasets/dataset.py to generate the word embedding in an online way

Run

• Update the path in train.sh, then

  sh train.sh

Test

• Update the path in the test.sh, then

  sh test.sh

Experiments

Evaluation results on benchmarks

• You can downlod the benchmark datasets from GoogleDrive.

Evaluate S-GTR with different settings

• Investigate the impact of different modules in S-GTR.

Plugging GTR in different STR baselines

• Plug GTR module into four representative types of STR methods.

1. Train GTR + CRNN model

   python GTR-plug/GTR-CRNN/train.py \
   --train_data data_lmdb_release/training --valid_data data_lmdb_release/validation \
   --select_data MJ-ST --batch_ratio 0.5-0.5 \
   --Transformation None --FeatureExtraction VGG --SequenceModeling BiLSTM --Prediction CTC
   --add_GTR True

   Test GTR + CRNN model.

   python GTR-plug/GTR-CRNN/test.py \
   --eval_data data_lmdb_release/evaluation --benchmark_all_eval \
   --Transformation None --FeatureExtraction VGG --SequenceModeling BiLSTM --Prediction CTC \
   --add_GTR True --saved_model saved_models/best_accuracy.pth

2. Train GTR + TRBA model.

   python GTR-plug/GTR-TRBA/train.py \
   --train_data data_lmdb_release/training --valid_data data_lmdb_release/validation \
   --select_data MJ-ST --batch_ratio 0.5-0.5 \
   --add_GTR True --Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn

   Test GTR + TRBA model

 python GTR-plug/GTR-TRBA/test.py \
--eval_data data_lmdb_release/evaluation --benchmark_all_eval \
--Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn \
--saved_model saved_model/best_accuracy.pth --add_GTR True

3. Train GTR + SRN model

   python GTR-plug/GTR-SRN/train.py \
   --train_data path-to-train-data --valid-data path-to-valid-data --add_GTR True

   Test GTR + SRN model

python GTR-plug/GTR-SRN/test.py \
--train_data --valid-data path-to-valid-data --add_GTR True

4. Train GTR + PRENBaseline model

   python GTR-plug/GTR-P-Base/train.py \
   --train_data path-to-train-data --valid-data path-to-valid-data --add_GTR True

   Test GTR + PRENBaseline model.

   python GTR-plug/GTR-P-Base/test.py \
   --train_data path-to-train-data --valid-data path-to-valid-data --add_GTR True

5. Train GTR + ABINet-LV model

   python GTR-plug/GTR-ABINet/main.py \
   --train_data path-to-train-data --valid-data path-to-valid-data --add_GTR True --config=configs/train_abinet.yaml 

   Test GTR + ABINet-LV model.

   python GTR-plug/GTR-ABINet/main.py \
   --valid-data path-to-valid-data  --add_GTR True --config=configs/train_abinet.yaml 

Issue

1. The train and test datasets are uploaded. The pretrain model will be uploaded and the training code for MT adaptive framework will be updated soon.

2. This code is for S-GTR and other GTR pluggin methods, and the pluggin models will be updated soon.

3. To facilitate interested workers to use our model to adapt to other language training, we will provide text guidance in README for other language recognition as soon as possible.

4. We will update the details of the visual recognition model, and provide guidance code to generate relevant feature maps for the question in issue.

Citation

Please consider citing this paper if you find it useful in your research.

@article{he2021visual,
  title={Visual Semantics Allow for Textual Reasoning Better in Scene Text Recognition},
  author={He, Yue and Chen, Chen and Zhang, Jing and Liu, Juhua and He, Fengxiang and Wang, Chaoyue and Du, Bo},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={36},
  year={2022}
}

Copyright

For research purpose usage only.