BabitMF / bmf

Cross-platform, customizable multimedia/video processing framework. With strong GPU acceleration, heterogeneous design, multi-language support, easy to use, multi-framework compatible and high performance, the framework is ideal for transcoding, AI inference, algorithm integration, live video streaming, and more.
https://babitmf.github.io/
Apache License 2.0
730 stars 60 forks source link

Add controlnet module #79

Closed xiaoweiw-nv closed 5 months ago

xiaoweiw-nv commented 6 months ago

The controlnet module uses a TensorRT-optimized controlnet implementation.

Currently, the prompt is read from a file by the text_module and passed to the controlnet_module, the graph looks like:

decode --> controlnet_module ---> encode
      |            ^
      |            |
      -------> text_module

To sync the prompt with the image, the decoder output is also sent to the text_module, once the decoder outputs EOF, the text_module will also generate EOF. However, in the controlnet_module, the propmt_queue is always empty.

To run the pipeline:

python test_controlnet.py

The input image in case you need it: bird

xiaoweiw-nv commented 6 months ago

Thanks @sfeiwong ! I will merge your modification into the pipeline and push another commit that contains everything.

xiaoweiw-nv commented 5 months ago

@sfeiwong New code pushed, empty queue problem solved, please review.