non-local module contains no information about temporal frames

psychopa4 / PFNL

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations

MIT License

138 stars 21 forks source link

non-local module contains no information about temporal frames #16

Closed oliverxudd closed 3 years ago

oliverxudd commented 3 years ago

Hi, pyschopa4, thanks for your good work! I have a question about your nonlocal module from fig4 in the paper. F in the figure is HW/(r^2) x HW/(r^2), which after softmax will multiply Y, so F contains spatial correlation only, because temporal channel is used as feature channel. So, How does this module model motion between temporal frames? It seems to me temporal weights remain same for all temporal frames because F contains no temporal info.

psychopa4 commented 3 years ago

A pixel in F could be computed from different spatial positions from X, which is why F contains temporal correlations. Please see the Figure below. nlc

oliverxudd commented 3 years ago

thanks for the clarification!