ewrfcas
commented
3 months ago Hi, we have explored related research in (https://yikai-wang.github.io/asuka/, https://arxiv.org/abs/2312.04831). To overcome this with prior completion is a good idea. However, I believe that the current algorithms for edge/sketch completion might not be able to keep up with the generative capabilities of diffusion (in fact, training a foundational edge completion model is not easy). For some relatively simple scenarios, providing completed simple edges or sketches may help stabilize the removal effect through models like ControlNet. Additionally, there could be many other factors influencing the uncontrollable outcomes of diffusion generation (such as noise or incomplete removal of shadows). Overall, using edge/sketch to ensure stability might be a method with a higher lower bound but also suffers from a lower upper bound (limited by the capacity of edge completion).
chl916185
看up主应该是做inpaint的大佬，想问一下，对于基于diffusion model的object removal，目前往往存在着mask区域生成不可控的问题，比如出现其他物体、或人等，问下大佬有没有研究过把重建prior的结果作为diffusion的guidance（如controlnet之类），让diffusion的inpainting变的更加稳定，即稳定生成背景？
大佬能否留个微信交流一下 现在这个方向似乎还是一个研究比较空白的地方