qiuyu96 / CoDeF

[CVPR 2024 Highlight] Official PyTorch implementation of CoDeF: Content Deformation Fields for Temporally Consistent Video Processing
https://qiuyu96.github.io/CoDeF/
Other
4.83k stars 385 forks source link

how to control style only in mask area? 怎么只对mask进行风格转换呢? #62

Open Yuhyeong opened 1 year ago

Yuhyeong commented 1 year ago

mask

I saw that some of the demos only performed style conversion on the divided objects and did not change other content, so I wanted to try it myself. But when I train the model and test it myself, I use the processed canonical, and the internal style will be transferred to the entire picture. I would like to ask how to control it? Could it be that after the canonical is also divided, only the divided content is processed? And the mask is not necessary, so what is the role of the mask in the entire training process?

我看demo里的有几个是只对分割后的物体进行了风格转换,没对其他的内容更改,于是想自己试一下

但是我在自己训练模型与测试的时候,使用处理完的canonical,内部的风格会迁移到整张图片里

想请问一下如何进行控制?难道是对canonical也进行分割后,只对分割内容处理吗?

而且mask也是不必须的,那在整个训练过程之中,mask起到的作用是什么?

canonical

Neither the paper nor the readme mentions the specific use of multiple canonical images. I saw in a tutorial on YouTube that multiple images can be used for foreground and background control? is this real? If so, which part should we start with?

论文里和readme里都没提到,多张canonical的具体使用,我在油管的一个tutorial里面看到可以使用多张图片来进行前景后景控制?这是真的吗?如果是的话,该从哪一部分下手?

ken-ouyang commented 1 year ago

如果训练的时候用到了masks,在训练中会自动生成N个model去分别训练各个Mask对应的部分(假设把视频分成N个部分)。这样在evaluate的时候会同时出N个Canonical。你可以对N个Canonical用对应的风格来实现不同部分转换不同风格。

Yuhyeong commented 1 year ago

如果训练的时候用到了masks,在训练中会自动生成N个model去分别训练各个Mask对应的部分(假设把视频分成N个部分)。这样在evaluate的时候会同时出N个Canonical。你可以对N个Canonical用对应的风格来实现不同部分转换不同风格。

大概明白了,感谢!

Becky-catherine commented 1 month ago

如果训练的时候用到了masks,在训练中会自动生成N个model去分别训练各个Mask对应的部分(假设把视频分成N个部分)。这样在evaluate的时候会同时出N个Canonical。你可以对N个Canonical用对应的风格来实现不同部分转换不同风格。

大概明白了,感谢!

请问您在迁移图像的时候遇到过重建的视频比较好,但是迁移后的视频静止的情况吗?