search

jingyuanli001 / RFR-Inpainting

The source code for CVPR 2020 accepted paper "Recurrent Feature Reasoning for Image Inpainting"
MIT License
355 stars 76 forks source link

Weird triangle artifact present in many images #39

Closed xiankgx closed 2 years ago

xiankgx commented 3 years ago

Dear @jingyuanli001 , hi, I've been training your model with the places2 large dataset for quite some time, over few million iterations, and I've been seeing the same weird triangle artifact appearing in many images.

Some modifications I've done are:

I think these artifacts appear when the holes are bigger. Have you encountered this problem? Please help, thank you.

all_seeing_eye all_seeing_eye2 all_seeing_eye3

jingyuanli001 commented 3 years ago

Hi, thanks for your question. In fact, this is also an issue we observed when developing the method (as well as testing a few other inpainting methods). Sometimes this issue is more significant. From my personal experience, there are three directions to work on to address the issue. First, you might try to tune the hyper-parameters with respect to perceptual loss and style loss, which might be helpful for addressing the artifact. Second, you might try to replace batch norm with some other normalization method to address this issue. Third, add an adversarial loss. This would require extra computation but can address the artifact a lot more effectively). We are not quite sure about the nature of this artifact though, but obviously by addressing these artifacts the evaluation results would be even better.

jingyuanli001 commented 3 years ago

This issue seems to be more significant when the masked region is large. It's possible that, because most of the masks used during training is not that large as some extreme cases, the model prefers to give good results on most of the training cases and ignores some extreme cases that are less common during training.

xiankgx commented 3 years ago

Thanks for your swift reply.

But this artifact is not during testing, but also during training. Like I said, I've increased the difficulty of the training by increasing the max stroke width and length by a constant multiplier.

Also, this artifact is more evident the longer we train. It is not very noticeable during the earlier iterations.

Somehow, the very same artifact appearing in many images is disturbing. It would be better if the artifacts look different.

Also, it somehow gives me the feeling of hitting some "local minimum". It is like you have a bubble between a soft sticker and a hard surface (mirror) and we try to remove the bubble by trying to move it out by pressing with our finger. It is possible it could appear that the trapped air can't go anywhere and further trying to fix the problem resulted in these "peaks".

xiankgx commented 3 years ago

I hope this explain what I felt with my last post.

Removing-Bubbles

xiankgx commented 3 years ago

Anyway, thanks for the swift reply. We will try our your suggestions.

jingyuanli001 commented 3 years ago

Thanks for your swift reply.

But this artifact is not during testing, but also during training. Like I said, I've increased the difficulty of the training by increasing the max stroke width and length by a constant multiplier.

Also, this artifact is more evident the longer we train. It is not very noticeable during the earlier iterations.

Somehow, the very same artifact appearing in many images is disturbing. It would be better if the artifacts look different.

Also, it somehow gives me the feeling of hitting some "local minimum". It is like you have a bubble between a soft sticker and a hard surface (mirror) and we try to remove the bubble by trying to move it out by pressing with our finger. It is possible it could appear that the trapped air can't go anywhere and further trying to fix the problem resulted in these "peaks".

That makes sense. But I am also feeling that, if you input the reconstructed images with artifacts into the model and re-inpaint the region with artifact, the artifact won't exist. Would this indicate that the last few iterations in RFR are actually less efficient, due to the overly large amount of information to process and parse?

xiankgx commented 3 years ago

Yeah, I agree that it would be possible to just detect these artifacts and redo a second round and have no problem with it.

Perhaps this is a result of iterative processing, when you reach the end, it is difficult to fix things without readjusting a major part of the other pixels. Just what I feel.

Please let us know if you have come across a solution. Anyway, I think the idea you presented in your paper is really novel and stands out from the rest. The result is really good too other than this artifact problem.

xiankgx commented 3 years ago

In other words, the errors are concentrated at the pixels in the final iteration, rather than evenly distributed across all the inpainted pixels.

jingyuanli001 commented 3 years ago

Yeah, I agree that it would be possible to just detect these artifacts and redo a second round and have no problem with it.

Perhaps this is a result of iterative processing, when you reach the end, it is difficult to fix things without readjusting a major part of the other pixels. Just what I feel.

Please let us know if you have come across a solution. Anyway, I think the idea you presented in your paper is really novel and stands out from the rest. The result is really good too other than this artifact problem.

Thanks! I will get back to you if we have more information.

nachifur commented 3 years ago

@jingyuanli001 Thank you very much for sharing your awesome work. I am wondering if you plan to share the Places2 weight?