search

TencentARC / T2I-Adapter

T2I-Adapter
3.43k stars 203 forks source link

What's the difference comparing T2I-Adapter with ControlNet? #2

Open eeyrw opened 1 year ago

eeyrw commented 1 year ago

They seems to have similar function.

florinshen commented 1 year ago

Same problem, they seems are concurrent work in one week...

kunalgoyal9 commented 1 year ago

Any update on this?

eeyrw commented 1 year ago

My finding is that no people pays attention to the difference... The ControlNet is getting quite popular due to prosperity of open source community but few people focus on T2l due to T2l acting as pure academic matters.

kunalgoyal9 commented 1 year ago

@eeyrw So there is no difference implementation wise? right?

eeyrw commented 1 year ago

@eeyrw So there is no difference implementation wise? right?

To be precise, there is no practical difference in aspect to application (to end user). But ControlNet and T2l use different network structure modification to control the image generation.

kunalgoyal9 commented 1 year ago

@eeyrw So there is no difference implementation wise? right?

To be precise, there is no practical difference in aspect to application (to end user). But ControlNet and T2l use different network structure modification to control the image generation.

Ah so only, difference is in the network structure. Thanks for the clarification :)

bonlime commented 1 year ago

@kunalgoyal9 the answers above are incorrect. There is one huge practical difference between T2I and ControlNet for end-user - the T2I doesn't slowdown image generation time. This is a significant factor if you want to use multiple controls. With CN generation time is 2-3x times slower, while using T2I has negligible impact on image generation speed

kunalgoyal9 commented 1 year ago

@bonlime Thanks for the clarification. Its without compromising the quality in results. if yes, is there any difference in architecture wise?

bonlime commented 1 year ago

I could answer your question, but instead I would suggest you to look through ContorlNet and T2I adapter papers first, it takes like 5 mins to find the difference yourself the quality of T2I is slightly worse in my experience

xinli2008 commented 3 months ago

In my opinion, The difference between the two lies in how the conditional image are handled and how the added feature maps are encoded.

  1. ControlNet uses a small convolutional network to achieve the effect of a VAE encoder([b, 3, 512, 512] -> [b, 4, 64, 64]), whereas T2I does not seem to have a dedicated process for handling conditions.

  2. ControlNet directly copies an SD encoder as the encoder for the condition map and uses convolution with zero initialization to ensure no noise is introduced into the training copy at the beginning of training. In contrast, T2I defines a series of small conv-resblock-downsample modules to achieve the same feature map size as the SD encoder.

  3. As for the difference in training time, both ControlNet and T2I need to pass through 2 different encoder. However, ControlNet has significantly more parameters for the condition map encoder compared to T2I's feature map encoder. Therefore, under the same conditions, SD's training time will be longer.

If there are any issues with what I said, please feel free to point.

Ang-Wei-Liang commented 2 months ago

May I have a summary of the difference in structure between a controlnet pipeline and a T2I pipeline in terms of inference? Or would a switch in models from controlnet to T2I be sufficient? Thank you.