How to use LAVIS for image tagging?

[GyrosOfWar]

Hi!

We'd like to use LAVIS for assigning tags to user-uploaded images. We're currently using it for creating captions, and that works great, but having tags would also be very helpful. I'm pretty much a complete newbie to ML. If you could provide a short code example on how to use LAVIS for assigning multiple keywords to images, that would be ideal. We're not really looking to train our own model. Otherwise, what can I read to find out more about this topic?

[dxli94]

This short demo shows how to tagging an image with a predefined set of labels. https://github.com/salesforce/LAVIS/blob/main/app/classification.py.

The core idea is to compute the image embedding, and measures how similar it is to the text embeddings of the text labels.

Hope this helps.

[GyrosOfWar]

Is there a way to also do that without predefined labels?

[dxli94]

The model won’t be able to know what are the tags of interest in that case.

获取 Outlook for iOShttps://aka.ms/o0ukef

---

发件人: Martin Tomasi @.> 发送时间: Sunday, March 12, 2023 5:27:38 PM 收件人: salesforce/LAVIS @.> 抄送: Dongxu @.>; Comment @.> 主题: Re: [salesforce/LAVIS] How to use LAVIS for image tagging? (Issue #189)

Is there a way to also do that without predefined labels?

― Reply to this email directly, view it on GitHubhttps://github.com/salesforce/LAVIS/issues/189#issuecomment-1465137725, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD4IK2A4DZKT7ZDVEDCCKKTW3WJIVANCNFSM6AAAAAAVWI2QEQ. You are receiving this because you commented.Message ID: @.***>

[jacquesfeng123]

model, visprocessors, = load_model_and_preprocess(name="blip_classification", model_type="base", is_eval=True, device=device)

Invalid number of classes provided, found -1

please help