search

makalo / CornerNet

tensorflow
120 stars 36 forks source link

The loss doesn't change #4

Open ckqsars opened 5 years ago

ckqsars commented 5 years ago

@makalo Because I only could train on the 1080ti GPU, I set the batch size is 2. However the loss is around 18 the focal_loss is around 36. I want to know the loss when you trian the model

makalo commented 5 years ago

I use eight TITAN-v to train. Loss can be reduced to 0.5. I provide single card and multi-card training in my code, but I haven't trained with single card, maybe it's a matter of training time. You can try multi-card training, because I have verified it on multi-card training.

ckqsars commented 5 years ago

@makalo I have used two TITAN to train. The loss was reduced to 3 when the step is around 170 thousand. How many steps you have trained when the loss was reduced to 0.5.

I also want to know the batch_size you set when you train.

Thank you very much

MaskVulcan commented 5 years ago

@ckqsars have you solve this problem?I get loss 3,too. And the result is very bad, I don't know how to reduce loss to 0.5

MaskVulcan commented 5 years ago

@makalo I have tryed the train.py and I see the debug image that saved, There are no predicted box but only Ground Truth, this means that it don't work at all. How can you train so well?