search

emilianavt / OpenSeeFace

Robust realtime face and facial landmark tracking on CPU with Unity integration
BSD 2-Clause "Simplified" License
1.46k stars 152 forks source link

About generating landmark heatmap offset groud truth label #35

Closed RobbertOnly closed 2 years ago

RobbertOnly commented 2 years ago

Hello Emiliana emilianavt: Very nice work for this project! And i have some problems about landmark heatmap offset label for training:

  1. How to generate groud truth?
  2. What is the loss function for landmark heatmap offset? I am looking forward to your reply.Thank you very much!
emilianavt commented 2 years ago

Hi, thank you!

  1. For generating the offset ground truth, you basically perform the inverse calculations given here. The result will look like a horizontal and vertical gradient from 0 to 1. If the ground truth landmark happens to be on exactly one of the pixel columns/rows after downscaling, it should have the value 0.5.
  2. Adaptive wing loss as given in model.py is used for this as well as the main heatmap.
GitZinger commented 1 year ago

what is the rationale for the x, y offset gradient heatmap? I did not see Adaptive wing loss using this. Is is a thing that you invented?

emilianavt commented 1 year ago

The offset maps allow the UNet part to upscale to a lower resolution than the input resolution while still giving high precision outputs. Without it, landmarks would have only 1/4 resolution per dimension.

It was described in both the ULSee document and the Real-Time Human Pose Estimation one linked here.