search

hysts / pl_gaze_estimation

MIT License
38 stars 13 forks source link
computer-vision gaze gaze-estimation pytorch pytorch-lightning

readme

pl_gaze_estimation

MIT License GitHub stars

Training code for gaze estimation models using MPIIGaze, MPIIFaceGaze, and ETH-XGaze.

https://user-images.githubusercontent.com/25161192/143683988-06ab6495-ae97-4201-b3f1-39dacf4bc69c.mp4

A demo is available in this repo.

Installation

pip install -r requirements.txt

For docker environment, see here.

Usage

The basic usage is as follows:

python train.py --configs /path/to/your/config.yaml

Multiple config files can be specified to set new variables or overwrite previously defined values. You can also overwrite previously defined values by listing the key/value pairs after the --options flags.

usage: train.py [-h] --configs [CONFIGS ...] [--options ...]

optional arguments:
  -h, --help            show this help message and exit
  --configs [CONFIGS ...]
                        Paths to config files.
  --options ...         Variables to overwrite. (optional)

MPIIGaze

test_id=0
python train.py --configs configs/examples/mpiigaze_lenet.yaml \
                --options EXPERIMENT.TEST_ID ${test_id} \
                          EXPERIMENT.OUTPUT_DIR exp0000/$(printf %02d ${test_id})

You need to download and preprocess the dataset before training by running the following commands:

bash scripts/download_mpiigaze_dataset.sh
python scripts/preprocess_mpiigaze.py --dataset datasets/MPIIGaze -o datasets/

(Additional packages are required to preprocess the dataset. See the code.)

MPIIFaceGaze

test_id=0
python train.py --configs configs/examples/mpiifacegaze.yaml \
                --options SCHEDULER.WARMUP.EPOCHS 3 \
                          EXPERIMENT.TEST_ID ${test_id} \
                          EXPERIMENT.OUTPUT_DIR exp0000/$(printf %02d ${test_id})

You need to download and preprocess the dataset before training by running the following commands:

bash scripts/download_mpiifacegaze_dataset.sh
python scripts/preprocess_mpiifacegaze.py --dataset datasets/MPIIFaceGaze_normalized -o datasets/

(Additional packages are required to preprocess the dataset. See the code.)

ETH-XGaze

python train.py \
    --config configs/examples/eth_xgaze.yaml \
    --options \
        VAL.VAL_INDICES "[1, 23, 24, 35, 38, 46, 58, 63, 70, 78]" \
        SCHEDULER.EPOCHS 15 \
        SCHEDULER.MULTISTEP.MILESTONES "[10, 13, 14]" \
        DATASET.TRANSFORM.TRAIN.HORIZONTAL_FLIP true \
        EXPERIMENT.OUTPUT_DIR exp0000

Docker Environment

docker-compose 1.29.2 is required.

Build

docker-compose build train

Train

docker-compose run --rm -u $(id -u):$(id -g) -v /path/to/datasets:/datasets train python train.py --configs /path/to/your/config.yaml

Results on ETH-XGaze dataset

All the results in the table below are of a single run. In these experiments, the data of subjects with id 1, 23, 24, 35, 38, 46, 58, 63, 70, and 78 were used as the validation data. The models were trained for 15 epochs with a multi-step learning rate schedule. The learning rate was multiplied by 0.1 at epochs 10, 13, and 14.

Model hflip GPU precision batch size optimizer lr weight decay training time val angle error val loss
EfficientNet-lite0 yes V100 32 64 Adam 0.0001 0 4h42m 5.330 0.06970
EfficientNet-b0 yes V100 32 64 Adam 0.0001 0 5h58m 5.139 0.06672
ResNet18 yes V100 32 64 Adam 0.0001 0 4h04m 4.878 0.06427
ResNet50 yes V100 32 64 Adam 0.0001 0 8h42m 4.720 0.06087
HRNet-18 yes V100 32 64 Adam 0.0001 0 21h56m 4.657 0.05937
ResNeSt26d yes V100 32 64 Adam 0.0001 0 8h02m 4.409 0.05678
RegNetY160 yes V100 32 64 Adam 0.0001 0 1d05h30m 4.377 0.05638
Swin-S yes V100 32 64 Adam 0.0001 0 1d00h08m 4.318 0.05629
HRNet-64 yes V100x8 16 64 AdamW 0.0008 0.05 3h11m 4.302 0.05523
ResNeSt269e yes V100x8 16 56 AdamW 0.0008 0.05 5h31m 4.045 0.05200

Related repos

References