Open muhark opened 7 months ago
I am not sure there has been other Llava models that don't use llama tokenizer no?
@ArthurZucker thank you for the fast response!
There are a few examples I know of:
Is there a strong reason for restricting the LlaVA Processor to using llama/clip?
There is none! It's just that we base it on usage! Could you link to the gemma based model? Feel free to open a PR
FYI @NielsRogge
Link to gemma-based model: Intel/llava-gemma-2b
I'm happy to open the PR :)
Feel free to do so and link it to this issue!
Hello, am just wondering how to append -200 image tokens to nay tokenizer and reuse LLavaProcessor?
Hey, you can simply use tokenizer.add_tokens(["toke1", "tok2", .........])
Feature request
Generalize the functionality in processing_llava.py to include other tokenizers and image processors.
Motivation
The current implementation of the LlaVA processor only accepts Llama as the tokenizer. Given the extensibility of the framework, this restriction should not be there.
Your contribution
I am happy to write the code and make the PR. We have done this for our own internal research code and there is an example file in our
llava-gemma-2b
model (link).