Include other tokenizers/image processors in Llava

[muhark]

Feature request

Generalize the functionality in processing_llava.py to include other tokenizers and image processors.

Motivation

The current implementation of the LlaVA processor only accepts Llama as the tokenizer. Given the extensibility of the framework, this restriction should not be there.

Your contribution

I am happy to write the code and make the PR. We have done this for our own internal research code and there is an example file in our llava-gemma-2b model (link).

[ArthurZucker]

I am not sure there has been other Llava models that don't use llama tokenizer no?

[muhark]

@ArthurZucker thank you for the fast response!

There are a few examples I know of:

• TinyLlaVA uses phi-2 (but requires users to install their package)
• We have just uploaded a Gemma-based LLaVA model

Is there a strong reason for restricting the LlaVA Processor to using llama/clip?

[ArthurZucker]

There is none! It's just that we base it on usage! Could you link to the gemma based model? Feel free to open a PR

FYI @NielsRogge

[muhark]

Link to gemma-based model: Intel/llava-gemma-2b

I'm happy to open the PR :)

[ArthurZucker]

Feel free to do so and link it to this issue!

[lucasjinreal]

Hello, am just wondering how to append -200 image tokens to nay tokenizer and reuse LLavaProcessor?

[ArthurZucker]

Hey, you can simply use tokenizer.add_tokens(["toke1", "tok2", .........])