search

X-PLUG / mPLUG-DocOwl

mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding
Apache License 2.0
1.12k stars 68 forks source link

Try to run sigclip model #63

Closed ViCtOr-dev13 closed 2 months ago

ViCtOr-dev13 commented 2 months ago

hello everyone , I 'm trying to run the tinychart sgclip model but I'm facing issues with this code : `model_path = "mPLUG/TinyChart-3B-768-siglip"

tokenizer, model, image_processor, context_len = load_pretrained_model( model_path, model_base=None, model_name=get_model_name_from_path(model_path), device="cuda" )` I have an error because it can't find tokenizer I suppose. image help me please.

zhangliang-04 commented 2 months ago

Hi @ViCtOr-dev13, mPLUG/TinyChart-3B-768-siglip is the vision encoder of mPLUG/TinyChart-3B-768 and it cannot be deployed separately. If your purpose is chart understanding, please try to load mPLUG/TinyChart-3B-768. It will load mPLUG/TinyChart-3B-768-siglip from huggingface hub automatically by default.

By the way, if you want to load the vision transformer locally, try to set mm_vision_tower in config.json of TinyChart-3B-768 to the local path of mPLUG/TinyChart-3B-768-siglip.

ViCtOr-dev13 commented 2 months ago

thanks a lot