Details on extraction of optical flow features

[Flyzanetothesky]

你好，我用作者提供的UCF101-split_1_tsn_flow_reference_bn_inception.caffemodel 提取某一个test video 的 optical flow images 的特征时，得到的结果(npy文件)与你所提供的差别很大，请问是什么原因呢？是不是因为我的 optical flow images 有问题呢？还是因为其他原因呢？请问你用TSN提取到的 optical flow images 分辨率是多少呢？需要做 resize 吗？我得到的optical flow images 宽高与视频相同，没有做resize，是这个问题导致提取到的特征与你的不相同吗？ 谢谢🙏

[HYPJUDY]

Hi @Flyzanetothesky , you mentioned that your extracted optical flow images features in the form of npy format files by UCF101-split_1_tsn_flow_reference_bn_inception.caffemodel are very different from mine.

What do you mean by the "difference"? I think the difference of the number in npy files is normal because the optical flow images may have some difference due to different version of opencv or other image extraction tools.

If I remember correctly, I also didn't resize images. And I think resizing or not and the difference I mentioned above will not have too much influence on the performance.

So I suggest you to train and test Decouple-SSAD with your features and compare the final results.

[Flyzanetothesky]

Hi @HYPJUDY , I'm very glad for your reply. I calculated the feature similarity (only use optical flow images and only use one test video) between you provided and mine, the result is 31.09%. But when i calculate the feature similarity between original optical flow images size (320 x 180) and resized (340 x 256), the result is 90.98%. So, you are right, resizing or not have little influence on the performance.

Today, I achieved the final results only use RGB images in main-stream network , the final results very similar to yours on "video_test_0000324.mp4". Results are as follows, the first fig is yours, the second is mine.

Same as you think, I think the result I got very different from yours when used optical flow because of the version of opencv and optical flow extraction tools, could you tell me which version of opencv you used?

In addition, the compute capability of my computer very poor, could you send your optical flow tools to me? then, I can used your provided models directly. my email : trytobeme@163.com. I am very eager to reproduce your paper. thank you very much.

[HYPJUDY]

Hi @Flyzanetothesky , How do you calculate the "feature similarity"? I extract frames and optical flow images refer to TSN. TSN use denseflow.

I used the default command in extract_optical_flow.sh, where new_width and new_height are set to 340 and 256 respectively. So I resized images to 340px*256px (sorry for my wrong memory in last comment). I don't remember the exactly version of opencv I used but I think any version that can run denseflow is OK.

I think there's no model of optical flow tools that can accelerate feature extraction process. Extraction of optical flow is indeed very time consuming. If you want to experiment on THUMOS14, I suggest you use my feature. Otherwise the number of 4 or more gpus is suggested for optical flow extraction.

[Flyzanetothesky]

Hi @HYPJUDY About "feature similarity". I extract optical flow feature only use "video_test_0000324.mp4" and then get 17 feature files, the shape of each feature files is (128, 1024). I ravel this feature and make its shape (131072, ) . Finally, I used cosine to calculate feature average similarity.

The results that I get is different when used CPU version of denseflow and GPU version. When I used CPU version of denseflow to extract the optical flow , the results was very different with yours. Today, I use GPU version of denseflow to extract the optical flow, the results is very similar to yours. By the way, the opencv version I used is 2.4.13.

Thanks for your suggestion, I make it !

[HYPJUDY]

Yep, I used GPU version of denseflow. I think CPU version would be super slow. Glad you make it.