search

Vision-CAIR / MiniGPT-4

Open-sourced codes for MiniGPT-4 and MiniGPT-v2 (https://minigpt-4.github.io, https://minigpt-v2.github.io/)
https://minigpt-4.github.io
BSD 3-Clause "New" or "Revised" License
25.38k stars 2.91k forks source link

What suggestions do you have for fine-tuning on a custom dataset? #258

Open MrWaterZhou opened 1 year ago

MrWaterZhou commented 1 year ago

Great job! I'm trying to fine-tune on a private dataset that consists of short descriptions about individuals, based on the stage-1 model, with the prompt "Write a short description for the main character.". Most cases are fine, but there are some results that do not meet expectations.

**1. It will generate a detailed description for some images

  1. Sometimes the generated text may lack key information, such as clothing or actions.
  2. Sometimes the text is just wierd ... like "I cannot see the image as it has been removed. Can you please provide the image so I can assist you better?"**

Do you guys have any suggestions, perhaps?

shiv6891 commented 1 year ago

Hi, Even I am trying to fine-tune for a custom image + prompt -> Answer/Explanation config. Any advice on this yet?