search

facebookresearch / pytorchvideo

A deep learning library for video understanding research.
https://pytorchvideo.org/
Apache License 2.0
3.26k stars 400 forks source link

Bug: Missing BatchNorm Layer in X3D #95

Open EClemMarq opened 3 years ago

EClemMarq commented 3 years ago

🐛 Bugs / Unexpected behaviors

Upon close inspection of the model graph for X3D_XS, X3D_S, X3D_M, & X3D_L (as downloaded from PyTorchHub), it appears that the downsampling convolution in the first ResBlock of the first ResStage (denoted as blocks.1.res_blocks.0.branch1_conv in the model's state_dict) is not followed by a batch normalization layer.

In all the other stages (ResStage[2]-ResStage[4]), there is a BN layer in the downsampling convolution. Is there a particular reason for excluding BN in the first stage?

haooooooqi commented 3 years ago

Hi @EClemMarq,

Could you please kindly share a the model stack you had from your side?

Thanks a lot!

EClemMarq commented 3 years ago

Hi @haooooooqi ! Thanks for looking into this. The code runs without errors, but I have a question about the model architecture. I don't know if I put this issue in the right category.

When I look at the model graph in TensorBoard, there is a discrepancy between the identity connection in ResBlock[0] of ResStage[1] and the identity connection in ResBlock[0] of all the other stages. I was wondering if there was a reason for the difference.

This first image shows ResBlock[0] from ResStage[2]. As marked by my blue arrow, this ResBlock (and the corresponding ResBlock[0] in ResStage[3] & ResStage[4]), all have a BatchNormalization layer on the identity connection. X3D_ResStage2_Graph_LI

The second image shows ResBlock[0] from ResStage[1]. As marked by my blue arrow, there is no BatchNormalization layer on this identity connection. X3D_ResStage1_Graph_LI

Is there a reason for this difference? Thanks!

EClemMarq commented 3 years ago

Hi, just following up on this. I'm building my own model using the pretrained weights available here, so I'm trying to make my model match exactly.

Thank you!

EClemMarq commented 3 years ago

Changing the category from a general question to a bug/unexpected behavior, since it seems quite unexpected.

This issue can also be seen without TensorBoard. Simply printing out model.summary() shows that there is not a BN layer in the identity connection of ResStage[1],ResBlock[0], even though this BN layer is applied elsewhere.

EClemMarq commented 2 years ago

Any news on this issue? For now, I'm just omitting this BN layer in my model so that it matches exactly and I can use the pretrained weights.