search

qiuyu96 / CoDeF

[CVPR 2024 Highlight] Official PyTorch implementation of CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
https://qiuyu96.github.io/CoDeF/
Other
4.83k stars 386 forks source link

Few questions about the paper #59

Open cats-food opened 1 year ago

cats-food commented 1 year ago

Hi, really thanks for the great work! I am new to the video processing and are confused about several places in the paper, really hope you can provide me some hints, thanks in advance :)

  1. what's the advantage of using hash table?
  2. Are the values in the 3D hash table trainable? How are their initial values determined?
  3. What are $\Delta x$, $\Delta y$ in Fig. 2? They don't seem to be mentioned in the main text. Do they represent residuals that will be added to the input coordinates (x, y)?
  4. Upon the optimization of the content deformation field, the canonical image $I_c$ is retrieved by setting the deformation of all points to zero. What does "setting the deformation of all points to zero" mean? Can I understand it as simply traversing all possible (x', y') coordinates and feeding them into the canonical MLP to obtain the canonical image?

Really appreciate your time, looking forward to your reply.

ken-ouyang commented 1 year ago

Hi,

  1. The hash table has high capability of representing the deformation.
  2. Yes. The features are learned during the training process.
  3. Yes. You can also use other transformation fields such as SO2 fields.
  4. Yes.