search

ertugrul-dmr / qwen2vl-captioner-gui

Apache License 2.0
8 stars 0 forks source link

its VRAM hungry ! #1

Open holditformehard opened 1 month ago

holditformehard commented 1 month ago

Great work to start with ! I am sure you will upgrade this marvel frequently To start with its a great UI. just need few tweaks.

1stly, I noticed that even after caption generation, offloading does not occur, hence eats up all the RAM. Could you make it more efficient?

2nd suggestion is that you please add batch processing.

ertugrul-dmr commented 1 month ago

For the first issue, I made some adjustments; however, offloading is not a viable solution since we would need to reload the model for every single generation. I created this repository for demonstration purposes and to help users get started. The batch version has already been implemented here: qwen2-vl-7b-captioner-relaxed-batch.