hrz2000
commented
1 week ago Thank you for your interest in this work!
Yes, this is a problem worth exploring, the method based on inpainting is currently unable to deal with operations with large differences in shapes, so when we construct the data, we use the original shape mask based on the same category label for inpainting (after training, it can be generalized to different categories), and the results are the same as the original shape.
This has both disadvantages and advantages, the disadvantage is that it is not possible to edit with a large difference in shape, because the object addition task is not constrained by the original shape, this problem of object replacement is mitigated to some extent, for example:
and the advantage is that it is also in line with the usual logic of instruction-based editing, i.e. the edited image should ideally maintain a similar shape to the object in the original image in most cases.
More advanced dataset construction methods can help further improve this problem.
mo230761
I am very interested in your research, and it is an excellent piece of work. But I tried using the BrushNet model to redraw data, but in the object replacement task, I found that BrushNet cannot complete inpainting when there is a large shape difference. Does this mean that the FreeBrench dataset also lacks data with large shape differences for object replacement? Has the FreeEdit model successfully handled object replacement in this regard?