search

Suikasxt / PMG

The repository of paper Personalized Multimodal Response Generation with Large Language Models
5 stars 0 forks source link

Issues with Generating High-Quality Keywords and Captions using Llama-2 and BLIP-2 #2

Open YiyanXu opened 1 month ago

YiyanXu commented 1 month ago

Hi!

I've noticed that the generation code of hard preference keywords and target keywords is not integrated into this repo. I tried to generate them using the prompts in the file prompts.py on the dataset ml-1m-latest, with Llama-2-7b-hf. However, the generated keywords are of low quality, and mostly repetitive, following the output example format "1. Keyword 1; 2. Keyword 2; ...".

Could you please provide any tips or tricks for generating high-quality keywords using Llama-2? Additionally, could you share the generation configuration for Llama-2 or directly provide the generated keywords for the dataset?

Furthermore, I also tried to caption movie posters in the dataset ml-1m-latest using BLIP-2. The generated captions seem to lack sufficient semantic information for the subsequent keyword-generation process. Could you also share any tips for generating better poster captions?

I truly appreciate any assistance you can provide and look forward to your response. Thank you!

Suikasxt commented 1 month ago

Do you means that the llm simply outputs "1. Keyword 1; 2. Keyword 2; ..."? I also encountered this promble while design the prompts. But after adjusting the order of examples and item infomation (from "The movies are..... The example of output is ....." to "The example of output is ..... The movies are ...."), it should be solved. It's not normal that you encounter this problem again using the prompts now. You can add my wechat (swm_sxt) for more information.

As for the captions from BLIP2, I think you can add some preface prompt, like "This is an image about ", and ask BLIP2 to continuation.

Another way to improve the keywords is to add introduction of the movies which is not provided in MovieLens, but you can access them from IMDB (https://blog.csdn.net/hhmy77/article/details/106389370). Web Crawler is not recommended because it is illegal.