search

vikhyat / moondream

tiny vision language model
https://moondream.ai
Apache License 2.0
5.03k stars 445 forks source link

Finetuning results in strange hallucination #109

Open enkie358 opened 3 months ago

enkie358 commented 3 months ago

Hello,

On finetuning the moondream2 model I keep getting strange hallucinations repeatedly, and unrelated to the content of the image, like

"The fireman had to use the ladder to get to the top floor of the building" or "once upon a time" or "there were two friends named Lily and Max who loved to explore the outdoors together. They were both fascinated by the world of insects and spent their days observing and learning about different types of"

Is there any advice on what might be causing this and how to reduce this?

Thanks!

vikhyat commented 3 months ago

Hard to comment in general, it depends on the fine tuning task, dataset size, hyperparameters used etc. Are you able to share any additional information about the fine tuning?