what is detailed operation of collapse window reshape and fc

[nianniana]

i am intereted in your great work. However, in the network architecture figure in the paper, i am confused about the detailed operation of "collapse window reshape and fc", would you please explain the procedure in detail. Thanks a lot!

[kenziyuliu]

Hi there,

Thanks for your interest! Have a look at the following code pointers:

1. For each MS-G3D block, we first expand the joint features by "unfolding" it (i.e. sliding temporal window, (N,C,T,V) --> (N,C,T,window_size*V)): https://github.com/kenziyuliu/MS-G3D/blob/master/model/msg3d.py#L43
2. We perform multi-scale graph convolutions on the spatial-temporal graphs (effectively larger graphs):
   • https://github.com/kenziyuliu/MS-G3D/blob/master/model/ms_gtcn.py#L55
   • https://github.com/kenziyuliu/MS-G3D/blob/master/model/ms_gtcn.py#L103
   • https://github.com/kenziyuliu/MS-G3D/blob/master/model/ms_gtcn.py#L106
3. The expanded features from unfolding the temporal windows are projected back to each frame with the FC layer (implemented with 1x1 conv, (N,C,T,window_size*V) --> (N,C,T,V):
   • https://github.com/kenziyuliu/MS-G3D/blob/master/model/msg3d.py#L54
   • https://github.com/kenziyuliu/MS-G3D/blob/master/model/msg3d.py#L65

(N=batch size, T=number of frames / number of windows, C=number of feature channels, V=number of nodes) This would be easier to understand if you try tracing the dimensions of the tensors. Hope this helps.