search

eric-ai-lab / MiniGPT-5

Official implementation of paper "MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens"
https://eric-ai-lab.github.io/minigpt-5.github.io/
Apache License 2.0
832 stars 51 forks source link

STAGE 1 TRAINING #47

Open srymaker opened 2 months ago

srymaker commented 2 months ago

Thank you for your great work! In the stage 1 training mentioned in the paper, is the input of llm images and text，because the description ‘After the pretraining stage, the model is capable of generating images for single text descriptions’ in the article is unimodal? And,What does ‘The Language Model (LLM) is then tasked with only generating these placeholders for text creation’ mean? Does the stage 1 llm only need to generate placeholders without other token embedding?

KzZheng commented 2 months ago

The unimodal here just means we only learn texts to image alignment in stage 1. I hope this figure can help you out.

Screenshot from 2024-04-25 13-36-00