Training

[siarez]

First, thank you for all the work and sharing the code. Currently, I'm using additional data to fine-tune the network, and I'm facing a strange situation where the training loss goes down, but the training pixel-MSE is goes up slightly. Here is an image of the curves.

• The model is being trained on a P100 Pascal GPU.
• I'm using PyTorch 1.0.
• I have experimented with freezing the first 3 hourglass modules with no luck.

I have been debugging the code for 4 days! And I'm still stuck. Any pointers is appreciated.

[siarez]

For reasons that I'm not entirely sure of yet, using a older version of the get_preds_fromhm from the commit history doesn't have this issue. Basically, the commit that address issue #32 is causing this issue.

I'm guessing the changes to get_preds_fromhm broke a symmetry somewhere.

[siarez]

Turns out the issue was how I was creating target heatmaps from ground truth landmarks. The transforms were confusing me. I'll submit a pool request with my function for heatmap creation which may be of interest for anyone who want to train the model in Pytorch.

[wqz960]

Hi @siarez have you trained your code on 300W. I want to train a model on 300W-2D dataset, have you try that?

[siarez]

I actually used my code to train (rather fine-tune) on our own internal dataset. Our internal dataset is also 2D. I'll try to create a public repository for training once my code is cleaned up. For now, you'll need to write your own fit and validation functions that use the create_target_heatmap function in my pull request to create target heat-maps from your ground truth landmarks.