Thermal Face is a machine learning model for fast face detection in thermal images. It was built for Fever, the contactless fever thermometer with auto-aim.
The face detection model is using TensorFlow Lite for optimal performance on mobile/edge devices. The recommended inference setup is a Raspberry Pi 4 Model B with a Coral USB Accelerator.
The following is an example for inference from Python on an image file using the compiled model thermal_face_automl_edge_fast_edgetpu.tflite
, downloaded from the latest release, and the Edge TPU API:
pip3 install Pillow
sudo apt-get install python3-edgetpu
from edgetpu.detection.engine import DetectionEngine
from PIL import Image
# One-time initialization:
face_detector = DetectionEngine('thermal_face_automl_edge_fast_edgetpu.tflite')
# Per-image detection:
image = Image.open('image.png').convert('RGB')
faces = face_detector.detect_with_image(image,
threshold=0.5
top_k=10,
keep_aspect_ratio=True,
relative_coord=False,
resample=Image.BILINEAR)
for face in faces:
# np.array([[left, top], [right, bottom]], dtype=float64)
print(face.bounding_box)
Alternatively, you can use the TF Lite API directly on the compiled model or, in the absence of an Edge TPU, on the uncompiled model thermal_face_automl_edge_fast.tflite
, which is also in the latest release.
The model is trained with Cloud AutoML using a face dataset that combines a large set of images in visible light from the WIDER FACE database and a smaller set of thermal images from the Tufts Face Database and the FLIR ADAS Dataset.
There are a total of 18,418 images and 164,915 face bounding box annotations in the combined dataset. The WIDER FACE set is large and diverse, but only contains visible-light images. The thermal images from the Tufts Face Database and FLIR ADAS Dataset are fewer and less diverse, so we mix the three sets before splitting them into training, validation, and test sets. The relative size of the test and validation sets are unusually small to achieve a better balance among the source datasets while still using a significant fraction of all available training data. The exact breakdown of images (and annotations) is as follows:
Training set | Validation set | Test set | |
---|---|---|---|
Tufts Face Database (IR) | 1,247 (1,247) | 155 (155) | 155 (155) |
Fraction of source images | 80% | 10% | 10% |
Fraction of combined | 7% (1%) | 39% (8%) | 39% (8%) |
FLIR ADAS (Faces) | 617 (854) | 77 (108) | 77 (99) |
Fraction of source images | 80% | 10% | 10% |
Fraction of combined | 3% (1%) | 20% (6%) | 20% (5%) |
WIDER FACE (Validation) | 2,898 (28,859) | 161 (1,611) | 161 (1,741) |
Fraction of source images | 90% | 5% | 5% |
Fraction of combined | 16% (18%) | 41% (86%) | 41% (87%) |
WIDER FACE (Training) | 12,870 (130,086) | 0 (0) | 0 (0) |
Fraction of source images | 100% | 0% | 0% |
Fraction of combined | 73% (81%) | 0% (0%) | 0% (0%) |
Combined | 17,632 (161,046) | 393 (1,874) | 393 (1,995) |
Fraction of combined sources | 96% (98%) | 2% (1%) | 2% (1%) |
Download the thermal images from the Tufts Face Database and upload them to Cloud Storage:
LOCATION="us-central1"
TDFACE_DIR="tufts-face-database"
TDFACE_BUCKET="gs://$TDFACE_DIR"
cd training
for i in $(seq 1 4)
do
curl -O http://tdface.ece.tufts.edu/downloads/TD_IR_A/TD_IR_A_Set$i.zip
curl -O http://tdface.ece.tufts.edu/downloads/TD_IR_E/TD_IR_E_Set$i.zip
done
mkdir $TDFACE_DIR
for f in TD_IR_*.zip
do
unzip $f -d $TDFACE_DIR/$(basename $f .zip)
done
gsutil mb -l $LOCATION $TDFACE_BUCKET
gsutil -m rsync -r $TDFACE_DIR $TDFACE_BUCKET
Create a dataset spec of the images in the AutoML format using the separately created bounding box annotations and upload it:
TDFACE_ANNOTATIONS="$TDFACE_DIR/bounding-boxes.csv"
TDFACE_AUTOML="tdface-automl.csv"
TDFACE_TRAINING_FRACTION=0.8
TDFACE_VALIDATION_FRACTION=0.1
TDFACE_TEST_FRACTION=0.1
curl https://github.com/maxbbraun/tdface-annotations/releases/latest/download/bounding-boxes.csv -o $TDFACE_ANNOTATIONS
python3 -m venv venv
. venv/bin/activate
pip3 install -r requirements.txt
python automl_convert.py \
--mode=TDFACE \
--tdface_dir=$TDFACE_DIR \
--tdface_bucket=$TDFACE_BUCKET \
--tdface_annotations=$TDFACE_ANNOTATIONS \
--training_fraction=$TDFACE_TRAINING_FRACTION \
--validation_fraction=$TDFACE_VALIDATION_FRACTION \
--test_fraction=$TDFACE_TEST_FRACTION \
--automl_out=$TDFACE_AUTOML
gsutil cp $TDFACE_AUTOML $TDFACE_BUCKET
Download the FLIR_ADAS_1_3.tar.*
files and separately created bounding box annotations and process them:
FLIR_ADAS_REPO="flir-adas-faces"
FLIR_ADAS_DIR="flir-adas-database"
TMP_FLIR_ADAS_DIR="/tmp/$FLIR_ADAS_DIR"
FLIR_ADAS_BUCKET="gs://$FLIR_ADAS_DIR"
FLIR_ADAS_ANNOTATIONS="$FLIR_ADAS_REPO/bounding-boxes.csv"
FLIR_ADAS_AUTOML="flir-adas-automl.csv"
FLIR_ADAS_TRAINING_FRACTION=0.8
FLIR_ADAS_VALIDATION_FRACTION=0.1
FLIR_ADAS_TEST_FRACTION=0.1
git clone https://github.com/maxbbraun/flir-adas-faces.git $FLIR_ADAS_REPO
cd $FLIR_ADAS_REPO
curl https://github.com/maxbbraun/flir-adas-faces/releases/latest/download/bounding-boxes.csv -o $FLIR_ADAS_ANNOTATIONS
mkdir $FLIR_ADAS_DIR
tar -C $FLIR_ADAS_DIR -xvf FLIR_ADAS_1_3.tar.001 --strip-components=1
python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
python flir_convert.py \
--input_dir=$FLIR_ADAS_DIR/train \
--output_dir=$TMP_FLIR_ADAS_DIR/train \
--output_csv=/dev/null
python flir_convert.py \
--input_dir=$FLIR_ADAS_DIR/val \
--output_dir=$TMP_FLIR_ADAS_DIR/val \
--output_csv=/dev/null
python flir_convert.py \
--input_dir=$FLIR_ADAS_DIR/video \
--output_dir=$TMP_FLIR_ADAS_DIR/video \
--output_csv=/dev/null
cd ..
. venv/bin/activate
python automl_convert.py \
--mode=FLIR \
--tdface_dir=$TMP_FLIR_ADAS_DIR \
--tdface_bucket=$FLIR_ADAS_BUCKET \
--tdface_annotations=$FLIR_ADAS_ANNOTATIONS \
--training_fraction=$FLIR_ADAS_TRAINING_FRACTION \
--validation_fraction=$FLIR_ADAS_VALIDATION_FRACTION \
--test_fraction=$FLIR_ADAS_TEST_FRACTION \
--automl_out=$FLIR_ADAS_AUTOML
gsutil cp $FLIR_ADAS_AUTOML $FLIR_ADAS_BUCKET
Download and upload the WIDER FACE dataset:
WIDERFACE_DIR="wider-face-database"
WIDERFACE_BUCKET="gs://$WIDERFACE_DIR"
mkdir $WIDERFACE_DIR
for f in WIDER_*.zip wider_*.zip
do
unzip $f -d $WIDERFACE_DIR/
done
gsutil mb -l $LOCATION $WIDERFACE_BUCKET
gsutil -m rsync -r $WIDERFACE_DIR $WIDERFACE_BUCKET
Create and upload the AutoML spec using the included bounding boxes:
WIDERFACE_TRAINING_AUTOML="widerface-training-automl.csv"
WIDERFACE_VALIDATION_AUTOML="widerface-validation-automl.csv"
WIDERFACE_TRAINING_TRAINING_FRACTION=1.0
WIDERFACE_TRAINING_VALIDATION_FRACTION=0.0
WIDERFACE_TRAINING_TEST_FRACTION=0.0
WIDERFACE_VALIDATION_TRAINING_FRACTION=0.9
WIDERFACE_VALIDATION_VALIDATION_FRACTION=0.05
WIDERFACE_VALIDATION_TEST_FRACTION=0.05
python automl_convert.py \
--mode=WIDERFACE \
--widerface_dir=$WIDERFACE_DIR/WIDER_train \
--widerface_bucket=$WIDERFACE_BUCKET/WIDER_train \
--widerface_annotations=$WIDERFACE_DIR/wider_face_split/wider_face_train_bbx_gt.txt \
--training_fraction=$WIDERFACE_TRAINING_TRAINING_FRACTION \
--validation_fraction=$WIDERFACE_TRAINING_VALIDATION_FRACTION \
--test_fraction=$WIDERFACE_TRAINING_TEST_FRACTION \
--automl_out=$WIDERFACE_TRAINING_AUTOML
python automl_convert.py \
--mode=WIDERFACE \
--widerface_dir=$WIDERFACE_DIR/WIDER_val \
--widerface_bucket=$WIDERFACE_BUCKET/WIDER_val \
--widerface_annotations=$WIDERFACE_DIR/wider_face_split/wider_face_val_bbx_gt.txt \
--training_fraction=$WIDERFACE_VALIDATION_TRAINING_FRACTION \
--validation_fraction=$WIDERFACE_VALIDATION_VALIDATION_FRACTION \
--test_fraction=$WIDERFACE_VALIDATION_TEST_FRACTION \
--automl_out=$WIDERFACE_VALIDATION_AUTOML
gsutil cp $WIDERFACE_TRAINING_AUTOML $WIDERFACE_BUCKET
gsutil cp $WIDERFACE_VALIDATION_AUTOML $WIDERFACE_BUCKET
Combine all AutoML dataset specs into one and upload it:
THERMAL_FACE_AUTOML="automl.csv"
MODEL_BUCKET="gs://thermal-face"
MODEL_NAME="thermal_face_automl_edge_fast"
gsutil mb -l $LOCATION $MODEL_BUCKET
rm -f $THERMAL_FACE_AUTOML
cat $TDFACE_AUTOML >> $THERMAL_FACE_AUTOML
cat $WIDERFACE_TRAINING_AUTOML >> $THERMAL_FACE_AUTOML
cat $WIDERFACE_VALIDATION_AUTOML >> $THERMAL_FACE_AUTOML
cat $FLIR_ADAS_AUTOML >> $THERMAL_FACE_AUTOML
gsutil cp $THERMAL_FACE_AUTOML $MODEL_BUCKET
Use Cloud AutoML Vision with the following options:
tdface_x_widerface_x_fliradas
$MODEL_BUCKET/$THERMAL_FACE_AUTOML
$MODEL_NAME
$MODEL_BUCKET/
Use Docker to compile the model for Edge TPU:
MODEL_FILE="$MODEL_NAME.tflite"
TPU_MODEL_FILE="${MODEL_FILE%.*}_edgetpu.${MODEL_FILE##*.}"
MODEL_DIR="$(pwd)/../models"
COMPILER_NAME="compiler"
OUT_DIR="/out"
gsutil cp $MODEL_BUCKET/**/*$MODEL_NAME*/model.tflite $MODEL_FILE
docker build . \
--no-cache \
--file=$COMPILER_NAME.Dockerfile \
--tag=$COMPILER_NAME \
--build-arg=MODEL_FILE=$MODEL_FILE \
--build-arg=OUT_DIR=$OUT_DIR
docker run \
--mount=type=bind,source=$MODEL_DIR,target=$OUT_DIR \
--rm \
$COMPILER_NAME
mv $MODEL_FILE $MODEL_DIR/