moondream模型的出现为我们提供了新的机遇。我对Image To Text节点进行了优化，使其能够根据输入的图像张量批次数量执行多次model.answer_question(img, query)，从而提高处理效率和输出质量。这对于需要针对不同图像进行批量提问的场景极为有用，例如询问图像中人物的性别、年龄等。
Hello! First and foremost, I would like to express my deepest respect and gratitude for your outstanding contributions to the ComfyUI community. Your work has not only pioneered interactions with language models and multimodal models but has also greatly enriched our user experience. Particularly, the moondream model, with its low memory footprint and high performance, has made complex interactions possible.
Motivation for Optimization
For a long time, the ComfyUI community has lacked nodes capable of effectively interacting with language models. Although the WD 1.4 Tagger was once a powerful tool for us, it tended to show a preference (or perhaps could be described as overfitting) when processing certain tags and lacked the ability to customize response directions. With the widespread adoption of the SDXL model, the prompts generated by WD 1.4 have become obsolete. Previously, GPT4V was a promising direction, but its cost and operational restrictions made it unsuitable for commercial environments. LLaVA and ShareGPT4V, despite their exceptional performance and ShareGPT4V's considerable support for Chinese, required too much memory overhead, forcing me to use dual 4090 cards to run my workflow (one for SDXL and one specifically for the ShareGPT4V model). I began searching for a more suitable solution until I encountered your project.
Solution and Implementation
The advent of the moondream model has provided us with new opportunities. I optimized the Image To Text node to perform multiple model.answer_question(img, query) operations based on the number of input image tensor batches, thereby improving processing efficiency and output quality. This is extremely useful for scenarios that require batch questioning for different images, such as inquiring about the gender, age, etc., of characters in images.
Conclusion
I believe this optimization will bring more efficient and accurate interaction methods to the ComfyUI community. Thank you again for your hard work, and I look forward to your valuable feedback. If you approve of my optimization, I plan to add more features in future versions, such as string concatenation and even the ability to perform pure text Q&A. After all, these are multimodal models, so they are also adept at pure text communication, enhancing the model's reusability.
中文版
尊敬的项目维护者，
您好！在此，我首先要对您在ComfyUI社区的卓越贡献表示深深的敬意和感谢。您的工作不仅开创了与语言模型及多模态模型交互的先河，也极大地丰富了我们的使用体验。特别是在moondream模型的帮助下，低显存开销与高性能的特点使得复杂交互成为了可能。
优化动机
长久以来，ComfyUI社区一直缺乏能够有效交互语言模型的节点。虽然WD 1.4 Tagger曾经是我们的强大工具，但它在处理某些tag时过于偏好（或许可以称之为过拟合），且无法自定义回答方向。随着SDXL模型的普及，WD 1.4产生的提示词已不再适用。此前，GPT4V虽然是一个有潜力的方向，但因其价格和运行限制而不适用于商业环境。LLaVA和ShareGPT4V虽然性能卓越，ShareGPT4V甚至对中文的支持也相当好，但是它对显存开销实在太大了，我不得使用4090双卡才能让工作流跑起来（一边跑SDXL、一边专门跑ShareGPT4V模型）。我开始寻找更适合的解决方案，直到遇到您的项目。
解决方案与实现
moondream模型的出现为我们提供了新的机遇。我对Image To Text节点进行了优化，使其能够根据输入的图像张量批次数量执行多次
model.answer_question(img, query)，从而提高处理效率和输出质量。这对于需要针对不同图像进行批量提问的场景极为有用，例如询问图像中人物的性别、年龄等。结语
我相信，这一优化将为ComfyUI社区带来更高效、更精准的交互方式。再次感谢您的辛勤工作，期待您的宝贵意见。如果你认可我的优化，我计划在后续版本中加入更多功能，如字符串拼接，甚至是实现纯文字问答的能力，毕竟它们本来就是多模态模型，所以也很擅长进行纯文字的交流，增强模型的复用性。
祝好， Logic
Eng Ver.
Dear Project Maintainer,
Hello! First and foremost, I would like to express my deepest respect and gratitude for your outstanding contributions to the ComfyUI community. Your work has not only pioneered interactions with language models and multimodal models but has also greatly enriched our user experience. Particularly, the moondream model, with its low memory footprint and high performance, has made complex interactions possible.
Motivation for Optimization
For a long time, the ComfyUI community has lacked nodes capable of effectively interacting with language models. Although the WD 1.4 Tagger was once a powerful tool for us, it tended to show a preference (or perhaps could be described as overfitting) when processing certain tags and lacked the ability to customize response directions. With the widespread adoption of the SDXL model, the prompts generated by WD 1.4 have become obsolete. Previously, GPT4V was a promising direction, but its cost and operational restrictions made it unsuitable for commercial environments. LLaVA and ShareGPT4V, despite their exceptional performance and ShareGPT4V's considerable support for Chinese, required too much memory overhead, forcing me to use dual 4090 cards to run my workflow (one for SDXL and one specifically for the ShareGPT4V model). I began searching for a more suitable solution until I encountered your project.
Solution and Implementation
The advent of the moondream model has provided us with new opportunities. I optimized the Image To Text node to perform multiple
model.answer_question(img, query)operations based on the number of input image tensor batches, thereby improving processing efficiency and output quality. This is extremely useful for scenarios that require batch questioning for different images, such as inquiring about the gender, age, etc., of characters in images.Conclusion
I believe this optimization will bring more efficient and accurate interaction methods to the ComfyUI community. Thank you again for your hard work, and I look forward to your valuable feedback. If you approve of my optimization, I plan to add more features in future versions, such as string concatenation and even the ability to perform pure text Q&A. After all, these are multimodal models, so they are also adept at pure text communication, enhancing the model's reusability.
Best regards, Logic