reference frames are not properly propagated

[dan64]

There are situations where the reference frames are not propagated properly.

For example using this reference frame (frame: 000)

The colored frame was

as you can see the top part of the man's coat on the left is colored light blue

the same problem happen if I consider the last frame (frame: 085)

Here the reference frame

and here the colored frame

in this case the nose of the man on the left is also blue.

I attached the full dataset used for this test: demo_test.zip

I noted that this problem is common in all the models using color propagation based on reference frames.

I hope that it will be possible to improve this part, because is not related to temporal color consistency. But to the more basic part that is related to color application.

Dan

[yyang181]

Hi @dan64,

Thank you so much for your interest in our work. I tested the case you provided and was able to replicate the color bleeding artifacts you mentioned. The issue likely stems from the fact that the VGG pretrained features used in BiSTNet are insufficient for accurately calculating correspondences between the input and reference frames. One way to verify this is by visualizing the features extracted by the VGG model from both the input and reference frames.

However, our other work, ColorMNet, addresses this issue more effectively by leveraging large pretrained visual models and memory-based mechanisms. We have attached the results from ColorMNet for your reference (only ref/000000.jpg was used as reference frame): colormnet.zip

The code and pretrained models for ColorMNet will be released soon.

[dan64]

This very interesting. ColorMNet will use PyTorch or TensorFlow ?

[yyang181]

PyTorch.