Test cbp issue

[guopei]

Hi @gy20073 ,

I am trying to test the output of the compact bilinear layer but can't get the result right. I have used this for training and the result is very good. This means the layer should work well. It must be something wrong about my code. It seems this is the best place to ask such questions, sorry if this causes any trouble...

As the paper says, the <psi(x), psi(y)> should approximate <x, y>^2. So my code is below:

import os
import caffe
import numpy as np

caffe.set_mode_gpu()
caffe.set_device(1)

model_prefix = 'only-cbp'

model_def = './' + model_prefix + '.prototxt'
model_weights = './' + model_prefix + '.caffemodel'

input2 = np.random.rand(1, 512, 1, 1)
input1 = np.random.rand(1, 512, 1, 1)

input2_ = np.outer(input2.squeeze(), input2.squeeze()).reshape(1,-1)
input1_ = np.outer(input1.squeeze(), input1.squeeze()).reshape(1,-1)

inner = np.inner(input2_, input1_)[0,0]

for i in range(0, 10):
    net = caffe.Net(model_def, model_weights, caffe.TEST)

    net.blobs['data'].data[...] = input2
    output = net.forward()
    output2 = output['bilinear'].squeeze().copy()

    net.blobs['data'].data[...] = input1
    output = net.forward()
    output1 = output['bilinear'].squeeze().copy()

    inner_ = np.inner(output2, output1)
    print(inner_, inner)

The prototxt file is as below:

name: "CUBCompactBilinearNet"
layer {
  name: "data"
  type: "Input"
  top: "data"
  input_param { shape: { dim: 1 dim: 512 dim: 1 dim: 1 } } 
}

layer {
  name: "bilinear_layer"
  type: "CompactBilinear"
  bottom: "data"
  bottom: "data"
  top: "bilinear"
  compact_bilinear_param {
    num_output: 8192
    sum_pool: true
  }
}

I have to pass something as model_def, so I just use a plain VGG net. No weight will be loaded though. A typical print result is as follows:

(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)
(1.0636638e+12, 16276.614808319961)

The inner product of cbp output seems to be higher than what it should be. I guess something must be wrong, my goal was to initialize the Net every iteration, but it doesn't seem to do so.

[gy20073]

Sorry for the confusion. Your code don't have any issues, the problem is that my implementation's output will scale up the inner product by 8192^2 (num_output^2).

The deeper cause of this confusing behavior is I misunderstood some behaviors of FFT libraries, but it won't affect the performance at all. Compact bilinear involves some large convolution operation, and I take advantage of the convolution theorem to accelerate it. I.e. A*B = iFFT(FFT(A).FFT(B)). However most of the FFT library we used now (cufft, kissfft) implement DTFT and thus a factor of 1/N is off compared to the continuous FFT. So the short story is that I forgot to divide DTFT(A) & DTFT(B) by num_output.

Although confusing, I think it might be better to not update this since it will not be compatible with the previously trained models and this issue doesn't affect performance. I will leave this issue open in case other people will encounter the same problem.

[guopei]

print(inner_ / inner / (8192*8192)) gives the output: 0.970258512713. Perfect!

Thanks for pointing out the reason!

[hubeihubei]

I want to ask do you get the result on test dataset ,I can't get the result on CUB dataset ,also it can't coverage and the loss is still about 5.3,I want to ask how you initilize the weight during finetuing .thanks for
you , I was troubled in the problem for weeks,please give me some help.

[gy20073]

have you followed the steps exactly as stated here?

https://github.com/gy20073/compact_bilinear_pooling/tree/master/caffe-20160312/examples/compact_bilinear https://github.com/gy20073/compact_bilinear_pooling/tree/master/caffe-20160312/examples/compact_bilinear

On May 24, 2018, at 12:18 AM, hubeihubei notifications@github.com wrote:

I want to ask do you get the result on test dataset ,I can't get the result on CUB dataset ,also it can't coverage and the loss is still about 5.3,I want to ask how you initilize the weight during finetuing .thanks for you , I was troubled in the problem for weeks,please give me some help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/gy20073/compact_bilinear_pooling/issues/5#issuecomment-391614354, or mute the thread https://github.com/notifications/unsubscribe-auth/AFv2jKdSct5hmThWGLHI5MtibcUiLLbLks5t1l7IgaJpZM4MCDNL.

[hubeihubei]

Thanks for replying me.what do you mean '' followed the steps"？Do you mean I have to do 3rd step "fine tune the last layer" using the VGG_ILSVRC_16_layers.caffemodel to initilize,then use the caffemodel trained in 3rd step to fine tune the whole network in 4th step .