search

salesforce / BLIP

PyTorch code for BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
BSD 3-Clause "New" or "Revised" License
4.76k stars 630 forks source link

code on text-video qa #65

Open cdqncn opened 2 years ago

cdqncn commented 2 years ago

Dear authors,

I was wondering if you could release the code on text-video qa(e.g., dataloader and how you process the videos).

Thanks!

LiJunnan1992 commented 2 years ago

Hi, you can refer to the code here for dataloading of text-video qa: https://github.com/salesforce/ALPRO. Thanks!

cdqncn commented 2 years ago

Thanks for your reply, this code is classification, I just want to learn how to generation for QA in BLIP.

LiJunnan1992 commented 2 years ago

We use the VQA model to generation answers: https://github.com/salesforce/BLIP/blob/48211a1594f1321b00f14c9f7a5b4813144b2fb9/models/blip_vqa.py#L85

To handle videos, we simply concatenate frame features and pass them to the text decoder.

BlueCat7 commented 2 years ago

@cdqncn Hi, have you reproduce the author result in zero-shot video QA. I tried to do it, but failed.

dipta007 commented 10 months ago

We use the VQA model to generation answers:

https://github.com/salesforce/BLIP/blob/48211a1594f1321b00f14c9f7a5b4813144b2fb9/models/blip_vqa.py#L85

To handle videos, we simply concatenate frame features and pass them to the text decoder.

@LiJunnan1992 By concat, do you mean, after getting the image (frame) encoding, all the encodings are concatenated and then passed that raw concatenated embeddings to the decoder? Thanks for such awesome repo by the way.