search

davidsandberg / facenet

Face recognition using Tensorflow
MIT License
13.82k stars 4.81k forks source link

which layers does train_softmax.py train? #1028

Open shashi438 opened 5 years ago

shashi438 commented 5 years ago

So I was wondering, if I use train_softmax.py on my own dataset (training set) of just two classes, along with --pretrained model of VGGFace2 (using Inception_resnet_v1), does the network train just the final fully connected layers, not updating the conv/pooling layer weights previously?

Also, I have removed all the center loss terms and values from the code, as center loss keeps on going higher and higher until it hits infinity. I think its because of just 2 classes. Now I don't have any problem like that

In a nutshell, I was wondering if right now, I'm just training a CNN's final FC layers, keeping the weights of all layers previously as constant? This is what I wish to do.

Thanks!

jetompki commented 5 years ago

I am encountering the same issue you had/have with the center loss value increasing until infinity. What did you change in order to get around this issue, and does it still actually train correctly?

I noticed that after center loss hit infinity, the training would eventually make Loss : nan Cl: nan, and other variables nan which has to be incorrect. This is with train_softmax.py.

shashi438 commented 5 years ago

So in case you are also doing few classes ( I did just 2) then you need to remove the center loss components from the code. Just look at where center loss is being calculated, and then comment it out, and also remove all resulting center loss additions into the total loss. I did this and had a good nw which performed as expected.

jetompki commented 5 years ago

Can you possibly show me the command line you used when training with softmax? I believe I've done what you said, however when I train it now. I have seemingly very high accuracy (1.00 99% of the time), and whenever my validate epoch occurs (every 5 epochs). It states:

Time 0:00 Loss Nan Xent Nan Accuracy Nan

Then also, the "Saving statistics" part freezes eventually. edit It seems it freezes eventually regardless of this.

I have two classes, both with 500 images.

shashi438 commented 5 years ago

I don't know what you mean by command. I used the normal command line of train_softmax.py as shown. (look at the parser arguments).

Also I didn't use validation function from code. I don't remember why but I think either the validate or the evalute function from code compares the image embeddings to some LFW dataset, and its complex.

I didn't have any problems with freezing so it has to do with your implementation.....(the dataset looks fine, but it might be very few images for a deep network like Resnet inception V1. Maybe try with a larger dataset and see if problem persists?)