search

BabitMF / bmf

Cross-platform, customizable multimedia/video processing framework. With strong GPU acceleration, heterogeneous design, multi-language support, easy to use, multi-framework compatible and high performance, the framework is ideal for transcoding, AI inference, algorithm integration, live video streaming, and more.
https://babitmf.github.io/
Apache License 2.0
773 stars 65 forks source link

Controlnet module #78

Closed xiaoweiw-nv closed 9 months ago

xiaoweiw-nv commented 9 months ago

The controlnet module uses a TensorRT-optimized controlnet implementation.

Currently, the prompt is read from a file by the text_module and passed to the controlnet_module, the graph looks like:

decode --> controlnet_module ---> encode
      |            ^
      |            |
      -------> text_module

To sync the prompt with the image, the decoder output is also sent to the text_module, once the decoder outputs EOF, the text_module will also generate EOF. However, in the controlnet_module, the propmt_queue is always empty.