Project structure

├── config/         # config for ICAL hyperparameter
├── data/
│   └── crohme      # CROHME Dataset
│   └── HME100k      # HME100k Dataset which needs to be downloaded according to the instructions below.
├── eval/             # evaluation scripts
├── ical               # model definition folder
├── lightning_logs      # training logs
│   └── version_0      # ckpt for CROHME dataset
│       ├── checkpoints
│       │   └── epoch=125-step=47375-val_ExpRate=0.6101.ckpt
│       ├── config.yaml
│       └── hparams.yaml
│   └── version_1      # ckpt for HME100k dataset
│       ├── checkpoints
│       │   └── epoch=55-step=175503-val_ExpRate=0.6924.ckpt
│       ├── config.yaml
│       └── hparams.yaml
├── .gitignore
├── README.md
├── requirements.txt
├── setup.py
└── train.py

Install dependencies

cd ICAL
# install project   
conda create -y -n ICAL python=3.7
conda activate ICAL
conda install pytorch=1.8.1 torchvision=0.2.2 cudatoolkit=11.1 pillow=8.4.0 -c pytorch -c nvidia
# training dependency
conda install pytorch-lightning=1.4.9 torchmetrics=0.6.0 -c conda-forge
# evaluating dependency
conda install pandoc=1.19.2.1 -c conda-forge
pip install -e .

Dataset Preparation

We have prepared the CROHME dataset and HME100K dataset in download link. After downloading, please extract it to the data/ folder.

Training on CROHME Dataset

Next, navigate to ICAL folder and run train.py. It may take 8~9 hours on 4 NVIDIA 2080Ti gpus using ddp.

# train ICAL model using 4 gpus and ddp on CROHME dataset
python -u train.py --config config/crohme.yaml

For single gpu user, you may change the config.yaml file to

gpus: 1

Training on HME100k Dataset

It may take about 48 hours on 4 NVIDIA 2080Ti gpus using ddp on HME100k dataset.

# train ICAL model using 4 gpus and ddp on hme100k dataset
python -u train.py --config config/hme100k.yaml

Evaluation

Trained ICAL weight checkpoints for CROHME and HME100K Datasets have been saved in lightning_logs/version_0 and lightning_logs/version_1, respectively.

# For CROHME Dataset
bash eval/eval_crohme.sh 0

# For HME100K Dataset
bash eval/eval_hme100k.sh 1

Experimental Results

Given the relative small size of the CROHME dataset, we conducted experiments with both the baseline CoMER and the proposed ICAL model using five different random seeds (7, 77, 777, 7777, 77777) under the same experimental conditions. The reported results are the averages and standard deviations of these five experiments.

Citation

@article{zhu2024ical,
      title={ICAL: Implicit Character-Aided Learning for Enhanced Handwritten Mathematical Expression Recognition}, 
      author={Jianhua Zhu and Liangcai Gao and Wenqi Zhao},
      year={2024},
      eprint={2405.09032},
      archivePrefix={arXiv},
      primaryClass={cs.CV},

}

Reference

• CoMER | arXiv
• BTTR | arXiv
• TreeDecoder
• CAN | arXiv