InfinityOutPainting example from README throws error

[Blubbaa]

The example for how to use the template classes, e.g. InfinityOutPainting from the README does not work and throws an error:

Thanks to @ShengmingYin and @thebestannie for providing a template example in InfinityOutPainting class (see the following gif)

Firstly, run python visual_chatgpt.py --load "ImageCaptioning_cuda:0,ImageEditing_cuda:1,VisualQuestionAnswering_cuda:2"

│   1260 │   │   # Load Basic Foundation Models                                                    │
│   1261 │   │   for class_name, device in load_dict.items():                                      │
│   1262 │   │   │   print(class_name)                                                             │
│ ❱ 1263 │   │   │   self.models[class_name] = globals()[class_name](device=device)                │
│   1264 │   │                                                                                     │
│   1265 │   │   # Load Template Foundation Models                                                 │
│   1266 │   │   for class_name, module in globals().items():                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: __init__() got an unexpected keyword argument 'device'

To me it seems there is a missing (potentially undocumented) piece on how to use these template classes, since they do not have a constructor with the device for initalization and thus fail to be used with the --load commandline parameter. Could you provide more information on how one should use this?

[Wang-Xiaodong1899]

Thanks, we are trying to fix it.

[woshialex]

looks like it is the problem with ImageEditing, once I removed it it can run. How can I add it back and fix the problem?

[Wang-Xiaodong1899]

Hi @Blubbaa, @woshialex, the code has been fixed. You can use the InfinityOutpainting by running python visual_chatgpt.py --load "Inpainting_cuda:0,ImageCaptioning_cuda:0,VisualQuestionAnswering_cuda:0". You can upload an image, and then say extend this image to 2048x1024.

BTW, we have incorporated the GroundingDINO and Segment-Anything for image detection and segmentation, by running python visual_chatgpt.py --load "Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0". You can upload an image, and then say find the cat in this image or segment the cat in this image.

Hope you enjoy it. Thanks~

[chivalrousS]

Hi @Blubbaa, @woshialex, the code has been fixed. You can use the InfinityOutpainting by running python visual_chatgpt.py --load "Inpainting_cuda:0,ImageCaptioning_cuda:0,VisualQuestionAnswering_cuda:0". You can upload an image, and then say extend this image to 2048x1024.

BTW, we have incorporated the GroundingDINO and Segment-Anything for image detection and segmentation, by running python visual_chatgpt.py --load "Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0". You can upload an image, and then say find the cat in this image or segment the cat in this image.

Hope you enjoy it. Thanks~

@Wang-Xiaodong1899 Hello, I also noticed that there was a problem with ImageEditing. There is still a problem. Once we added imageediting_cuda :0 to the command, there was an error.

[Wang-Xiaodong1899]

hi @chivalrousS, we have changed the ImageEditing to a template model, so this ImageEditing model should not be used in the command line. When you have imported "Text2Box, Segmenting, Inpainting, and ImageCaptioning", this imageEditing model will be automatically imported.

So, if you want to use ImagEditing, please use this command: python visual_chatgpt.py --load "Text2Box_cuda:0,Segmenting_cuda:0,Inpainting_cuda:0,ImageCaptioning_cuda:0". Thanks~

[chivalrousS]

@Wang-Xiaodong1899 Thank you very much,i get it.