Hello from ControlNet

[lllyasviel]

Hello instruct-pix2pix,

This is team of ControlNet. We trained a controlnet model with ip2p dataset here. See the section "ControlNet 1.1 Instruct Pix2Pix".

Rightnow the behavior of that model is different but the performance is similar to official ip2p. Do you think it is possible to improve the robustness with better datasets or some other stratagies?

[PospoSK]

Is there any tutorial, how it works now in A1111? Thx.

[andreemic]

Do you have any more specific observations of the performance compared to the original InstructPix2Pix?

We've fine-tuned both ControlNet and ip2p on a 50k hand-filtered task-specific dataset and might have some insights. @lllyasviel

[peterwilli]

I trained my own controlnet instructpix2pix a few weeks prior. Idk if it helps, but here is a demo: https://colab.research.google.com/drive/1kG7rLHGC4rGrLwGIGKiNsFZI7vh-_fB4?usp=sharing

It's trained on the full clip-filtered dataset.

[timothybrooks]

Hi @lllyasviel, awesome work on ControlNet :) I believe there is plenty of room to improve the robustness of instruct-pix2pix, in particular by improving the training dataset (generating better captions/edit instructions, better models/methods for generating pairs of images, and better dataset filtering). There are probably other strategies to improve it as well.