(huggingface sometimes may not work with safari, use chrome)
Demo
News
(Can run on 8GB memory GPU within 20S!)
Main Pipeline
Reasoning Details
The text to image model is conrolnet with canny from diffuser.
Please find installation instructions in install.md.
export OPENAI_KEY=[YOUR KEY HERE]
python main.py --image_src [image_path] --out_image_name [out_file_name]
If your GPU memory smaller than 8 GPB.
python main.py --image_caption_device cpu --semantic_segment_device cpu
If you have no GPU available.
python main.py --image_caption_device cpu --semantic_segment_device cpu --dense_caption_device cpu --contolnet_device cpu
like
python main.py --image_src "examples/3.jpg" --out_image_name "output/3_result.jpg"
Note: If you have GPU card with larger memory than 15GB. Set all device to GPU for fast inference.
The generated text and image are show in "output/".
Note: Use GPT4 for good result as GPT3.5 miss the position information sometime.
python main_gradio.py
If you have GPU Memory larger than 20GB. Use device='cuda' as default.
A dog sitting on a porch with a bike. | |||
Input | BLIP2 Image Caption | GRIT Dense Caption | Semantic Segment Anything |
The final generated paragraph with ChatGPT is:
This image depicts a black and white dog sitting on a porch beside a red bike. The dense caption mentions other objects in the scene, such as a white car parked on the street and a red bike parked on the side of the road. The region semantic provides more specific information, including the porch, floor, wall, and trees. The dog can be seen sitting on the floor beside the bike, and there is also a parked bicycle and tree in the background. The wall is visible on one side of the image, while the street and trees can be seen in the other direction.
Method | Trainable Parameter | Running Time | IR@1 | TR@1 |
---|---|---|---|---|
Image-text | 230M | 9H | 43.8 | 33.2 |
Generated Paragraph-text | 0 | 5m | 49.7 | 36.1 |
Interesting, we find compress image into paragraph. The retrieval result is even better than use source image.
If you have more suggestions or functions need to be implemented in this codebase, feel free to drop me an email awinyimg dot gmail dot com or open an issue.
This work is based on ChatGPT, Edit_Anything, BLIP2, GRIT, OFA,Segment-Anything, Semantic-Segment-Anything, ControlNet.