How to compute volume mean from list?

[MichaelXin]

Hi,

I am trying to train C3D from scratch. But I met a problem when trying to compute the volume mean.

So... what is the meaning of length and sampling_rate in compute_volume_mean_from_list.cpp? And how to set their values ?

For example, my input_chunk_list is like this:

/media/michael/F/data/UCF-101/UCF-101-frames/v_ApplyEyeMakeup_g08_c01 1 0
/media/michael/F/data/UCF-101/UCF-101-frames/v_ApplyEyeMakeup_g08_c01 17 0
/media/michael/F/data/UCF-101/UCF-101-frames/v_ApplyEyeMakeup_g08_c01 33 0
......

And I write a script to commpute volumn mean: GLOG_logtostderr=1 /home/michael/workspace/C3D/build/tools/compute_volume_mean_from_list.bin /home/michael/workspace/C3D/examples/ARCH/train_01_for_com_mean.lst 16 128 171 1 /home/michael/workspace/C3D/examples/ARCH/mean_train_16_128_171.binaryproto

the length is set to be 16 and the sampling_rate is set to be 1, because I found the new_length in /c3d_finetuning/c3d_ucf101_finetuning_train.prototxt is 16 .

name: "DeepConv3DNet_ucf101_finetuning" layers { name: "data" type: VIDEO_DATA top: "data" top: "label" image_data_param { source: "train_01.lst" use_image: true mean_file: "train01_16_128_171_mean.binaryproto" use_temporal_jitter: false batch_size: 30 crop_size: 112 mirror: true show_data: 0 new_height: 128 new_width: 171 new_length: 16 shuffle: true } }

But it seems that something is wrong. So I think my setting 16 128 171 1 (length height width sampling_rate) when calling compute_volume_mean_from_list.bin is wrong.

Could you give me some suggestions about this question?

Thank you very much!

[dutran]

The command and the input arguments look correct. I think the problem may come from the list file. Some (or at least one) of your input clips (a line in the list file) cannot be read. As shown in your log file, your datum size is 0. But it seems a few clips at the beginning are successfully read. You can check, this is true if you find "Starting Iteration" in your output log.

[MichaelXin]

Hi Du， Thank you for your prompt reply！ I probably found the reason.

The collapse point is at ：

CHECK_EQ(size_in_datum, data_size) << "Incorrect data field size " << size_in_datum;

in compute_volume_mean_from_list.cpp.

When the caffe/util/image_io.cpp read this line:

/media/michael/F/data/UCF-101/UCF-101-frames/v_ApplyEyeMakeup_g09_c04 177 0

with length=16 and sampling_rate=1, image_io.cpp try to read images from 000177.jpg to 000192.jpg, however, v_ApplyEyeMakeup_g01_c04.avi have only 191 frames. So, the ReadImageSequenceToVolumeDatum() will return false. And datum.data().size() is 0;

However, in compute_volume_mean_from_list.cpp, (from Line 88 to 92),

ReadImageSequenceToVolumeDatum(frm_dir.c_str(), start_frm, label, length, height, 
                                     width, sampling_rate, &datum);
const string& data = datum.data();
size_in_datum = datum.data().size(); //
CHECK_EQ(size_in_datum, data_size) << "Incorrect data field size " << size_in_datum;

CHECK_EQ(size_in_datum, data_size) is not passed, because size_in_datum=0.

I try to revise the code：

CHECK_EQ(size_in_datum, data_size) << "Incorrect data field size " << size_in_datum;

into this:

if(size_in_datum !=  data_size) {
      continue;
}

Then it works. But I`m not sure if this is right.

[dutran]

Hi @MichaelXin, your revised code just help you to skip the missing clips, but not solve your problem. I think for computing volume mean, it does not really affect the mean much, e.g. you have many clips and missing some of them does not significantly change the mean. However, you need to fix you list file for later usage (training, fine-tuning).

There is one difference, and I notice that it may be your problem. When you use frames as inputs, frame number starting from 1-n, but if you use video, frame number starting from 0 to (n-1), so caused your problem of missing clips at the end (boundary case).

I am closing this issue for now as the problem comes from the list file.

[MichaelXin]

Thank you for your kind remind! That's really helpful~