Not getting good results while using my own Condition Network

[asadabbas09]

I'm trying to use my own condition network and visualize some neurons in conv5_2layer. The network had different layer names so I changed self.fc_layers and self.conv_layers in sampling_class.py. I also changed 3_hidden_conditional_sampling.sh accordingly as well.

I tried sweeping across epsilon1, epsilon2, epsilon3 and learning rates parameters and performed 5000 iterations, but network fails to generate good images with high output probabilities.

I'm not sure if I am sweeping across right parameters, some of the parameters that I have tried are:

lr=(0.0005 0.005 0.05 1) 
epsilon1=(5 1 1e-1 1e-3 1e-7 1e-11 1e-15)
epsilon2=(0.00001 0.0001 0.05 0.5 1 2)
epsilon3=(5 1 1e-1 1e-3 1e-7 1e-11)

Is there anything else I need to change to get it working for my own condition network or How would you recommend to proceed in this case?

[anguyen8]

@asadabbas09 : Thanks for your question! What is the architecture of your condition network? (e.g. GoogleNet or something?)

[asadabbas09]

It's a VGG16 network trained for face recognition.

[anguyen8]

@asadabbas09 : usually, for hidden neurons it's very easy to make them reach high activations / probabilities. Are you able to get high probabilities at all with e3=0?

[asadabbas09]

@anguyen8 I've tried using e3=0 as well. but still, condition_unit probabilities aren't getting high. Most of the time best_unit repeats a pattern and if I set condition_unit to best_unit according to the pattern (248 in given case) I do get some high activations, but generally, there's no convergence.

for condition_unit 55

step: 0479   max:   19 [0.49]    obj:   55 [0.00000000]  norm: [0.00]
step: 0480   max:   19 [0.52]    obj:   55 [0.00000000]  norm: [0.00]
step: 0481   max:   19 [0.58]    obj:   55 [0.00000000]  norm: [0.00]
step: 0482   max:   21 [0.54]    obj:   55 [0.00000000]  norm: [0.00]
step: 0483   max:   21 [0.78]    obj:   55 [0.00000000]  norm: [0.00]
step: 0484   max:   21 [0.82]    obj:   55 [0.00000000]  norm: [0.00]
step: 0485   max:   21 [0.88]    obj:   55 [0.00000000]  norm: [0.00]
step: 0486   max:   21 [0.81]    obj:   55 [0.00000000]  norm: [0.00]
step: 0487   max:   21 [0.54]    obj:   55 [0.00000000]  norm: [0.00]
step: 0488   max:  248 [0.40]    obj:   55 [0.00000000]  norm: [0.00]
step: 0489   max:  248 [0.51]    obj:   55 [0.00000000]  norm: [0.00]
step: 0490   max:  248 [0.69]    obj:   55 [0.00000000]  norm: [0.00]
step: 0491   max:  248 [0.83]    obj:   55 [0.00000000]  norm: [0.00]
step: 0492   max:  248 [0.84]    obj:   55 [0.00000000]  norm: [0.00]
step: 0493   max:  248 [0.85]    obj:   55 [0.00000000]  norm: [0.00]
step: 0494   max:  248 [0.89]    obj:   55 [0.00000000]  norm: [0.00]
step: 0495   max:  248 [0.91]    obj:   55 [0.00000000]  norm: [0.00]
step: 0496   max:  248 [0.91]    obj:   55 [0.00000000]  norm: [0.00]
step: 0497   max:  248 [0.91]    obj:   55 [0.00000000]  norm: [0.00]
step: 0498   max:  248 [0.89]    obj:   55 [0.00000000]  norm: [0.00]
step: 0499   max:  248 [0.88]    obj:   55 [0.00000000]  norm: [0.00]
step: 0500   max:  248 [0.87]    obj:   55 [0.00000000]  norm: [0.00]

If I set condition_unit to 248, again I get similar pattern but some samples for 248th neuron at the end.

step: 0479   max:   19 [0.49]    obj:  248 [0.00000011]  norm: [0.00]
step: 0480   max:   19 [0.52]    obj:  248 [0.00000035]  norm: [0.00]
step: 0481   max:   19 [0.58]    obj:  248 [0.00000137]  norm: [0.00]
step: 0482   max:   21 [0.54]    obj:  248 [0.00001180]  norm: [0.00]
step: 0483   max:   21 [0.78]    obj:  248 [0.00012804]  norm: [0.00]
step: 0484   max:   21 [0.82]    obj:  248 [0.00143779]  norm: [0.00]
step: 0485   max:   21 [0.88]    obj:  248 [0.02325740]  norm: [0.00]
step: 0486   max:   21 [0.81]    obj:  248 [0.11571622]  norm: [0.00]
step: 0487   max:   21 [0.54]    obj:  248 [0.29134911]  norm: [0.00]
step: 0488   max:  248 [0.40]    obj:  248 [0.40160143]  norm: [0.00]
step: 0489   max:  248 [0.51]    obj:  248 [0.50645238]  norm: [0.00]
step: 0490   max:  248 [0.69]    obj:  248 [0.68800771]  norm: [0.00]
step: 0491   max:  248 [0.83]    obj:  248 [0.82597047]  norm: [0.00]
step: 0492   max:  248 [0.84]    obj:  248 [0.84389746]  norm: [0.00]
step: 0493   max:  248 [0.85]    obj:  248 [0.85142034]  norm: [0.00]
step: 0494   max:  248 [0.89]    obj:  248 [0.89233971]  norm: [0.00]
step: 0495   max:  248 [0.91]    obj:  248 [0.90517473]  norm: [0.00]
step: 0496   max:  248 [0.91]    obj:  248 [0.90765899]  norm: [0.00]
step: 0497   max:  248 [0.91]    obj:  248 [0.90551907]  norm: [0.00]
step: 0498   max:  248 [0.89]    obj:  248 [0.89001757]  norm: [0.00]
step: 0499   max:  248 [0.88]    obj:  248 [0.87751538]  norm: [0.00]
step: 0500   max:  248 [0.87]    obj:  248 [0.87093037]  norm: [0.00]

I've also tried couple of other caffe models, only one of them (having two outptut classes) converges and that for output neurons only. I must be missing something, as places205 and bvlc_reference_caffenet models works for many different values of e1, e2, e3 or lr.

[anguyen8]

@asadabbas09 : One of the known problems is optimization becomes less effective when a neuron is in a deep layer (e.g. in ResNet). It's harder to get such neuron highly activated... however, this case VGG is not that deep. I guess it must be something specific to this model that you're using.