search

salesforce / LAVIS

LAVIS - A One-stop Library for Language-Vision Intelligence
BSD 3-Clause "New" or "Revised" License
9.75k stars 955 forks source link

How to run InstructBLIP #313

Open ouhenio opened 1 year ago

ouhenio commented 1 year ago

Hello!

I'm trying to run Vicuna InstructBLIP, but sadly, I can't make it work.

I installed LAVIS directly from your repo following the step 3 of the installation guide, and I'm using the following code:

import torch
from lavis.models import load_model_and_preprocess

device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

model, vis_processors, _ = load_model_and_preprocess(
    name="blip2_vicuna_instruct", 
    model_type="vicuna7b", 
    is_eval=True, 
    device=device
)

With which I get the following error:

File "~/.../LAVIS/lavis/models/blip2_models/blip2_vicuna_instruct.py", line 82, in __init__
    self.llm_tokenizer = LlamaTokenizer.from_pretrained(llm_model, use_fast=False, truncation_side="left")

...

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './llm/vicuna-7b'. Use `repo_type` argument if needed.

I guess this happens because inside the blip2/blip2_instruct_vicuna7b.yaml configuration llm_model is set to "./llm/vicuna-7b".

I tried to fix this by manually downloading the model from the pretrained value present in said yaml, and create a ./llm/vicuna-7b folder to store it but it still fails.

Any ideas on how can I run this?

ouhenio commented 1 year ago

Ok I'm a dumb dumb, this was discussed and apparently solved in another issue.

ouhenio commented 1 year ago

In case anyone has trouble with this in the future, here are the steps I took to run InstructBLIP-7B:

Download Llama

This can be done quickly via transformers

import os
from transformers import LlamaTokenizer, LlamaForCausalLM

save_folder = "/path/to/llms/folder"

tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
tokenizer.save_pretrained(os.path.join(save_folder, "tokenizer"))

model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")
model.save_pretrained(os.path.join(save_folder, "model"))

Note that we are manually saving both the tokenizer and model in a folder. This can also be achieved with the cache_dir flag.

Download Alpaca

Now that you have Llama locally, you'll need to add the delta weights to convert this into Alpaca.

This is done installing FastChat and then following the Vicuna 7b instructions.

When you follow FastChat's instructions make sure that --base-model-path matches with the save_folder value you used in the Download Llama section.

Install LAVIS

You need to install LAVIS manually. This won't work with the pip version.

git clone https://github.com/salesforce/LAVIS.git
cd LAVIS
pip install -e .

Now go into LAVIS/lavis/configs/models/blip2/blip2_instruct_vicuna7b.yaml and change the llm_model attribute with the same one you used in the --target-model-path flag in the Download Alpaca section.

Use InstructBLIP

import torch
from lavis.models import load_model_and_preprocess

device = torch.device("cuda") if torch.cuda.is_available() else "cpu"

model, vis_processors, _ = load_model_and_preprocess(
    name="blip2_vicuna_instruct", 
    model_type="vicuna7b", 
    is_eval=True, 
    device=device
)

This code snippet should work, meaning you are ready to play with InstructBLIP. :blush:

If you want to use the 13B version, follow the same steps but change 7b to 13b accordingly.

lucastack commented 1 year ago

nice, it works!

Don-Chad commented 1 year ago

Guys what size GPU's are you running it at? My does hangs at.. 'Loading model...'; I have a 3090 24GB. Should be enough, I hope?

ouhenio commented 1 year ago

@Don-Chad I'm using a 3090 with 24GB too, and at least the 7B model worked flawlessly. I haven't tried with bigger ones though.

Rrrila commented 1 year ago

I'm complete newbie on python, but, would it be possible you help me on running this on a bigger model… Or how can I run this, let's say I want to use falcon 30b model (which is amazing results as an llm chat)

My goal is to be able to use a big llm to throw some questions about a specific image using blip2.

unoriginalscreenname commented 1 year ago

Do you really have to go through all of those steps to get the instructBLIP to work? I'm confused why the huggingface model card doesn't list any of that or mention LAVIS. (https://huggingface.co/Salesforce/instructblip-vicuna-7b)

Also, for 7b this seems like a huge download. Maybe I'm used to using the quantized models from TheBloke, but this seems to big to run on my 24g 3090. I can't say I fully understand what's going on here from a technical standpoint, but is there a way to quantize this? Is there an easier, more out-of-the-box way to get this up and running?

DiWHNJ commented 1 year ago

Thank you very much for your explanation. After I followed the steps you mentioned and started using instructBLIP, there were two errors as follows:

Traceback (most recent call last):
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 1354, in do_ Open
h. Request (Req. get_method(), Req. selector, Req. data, headers,
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/http/client. py", line 1256, in request
Self_ Send_ Request (method, URL, body, headers, encode_chunked)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/http/client. py", line 1302, in_ Send_ Request
Self. endheaders (body, encode_chunked=encode_chunked)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/http/client. py", line 1251, in endheaders
Self_ Send_ Output (messagebody, encode_chunked=encode_chunked)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/http/client. py", line 1011, in_ Send_ Output
Self. send (msg)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/http/client. py", line 951, in send
Self. connect()
File "/yq/software/anaconda3/envs/navis/lib/Python 3.8/http/client. py", line 1425, in connect
Self. lock=self_ Context. wrap_ Socket (self. lock,
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/ssl. py", line 500, in wrap_ Socket
Return self.sslsocket_ Class_ Create（
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/ssl. py", line 1040, in_ Create
Self.do_ Handshake()
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/ssl. py", line 1309, in do_ Handshake
Self_ Sslobj.do_ Handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "useiblip. py", line 7, in<module>
Model, vis_ Processors, _= Load_ Model_ And_ Preprocess（
File "/yq/dmz/IntrutBLIP/LAVIS main/lavis/models/__init__. py", line 200, in load_ Model_ And_ Preprocess
Model=model_ Cls. from_ Pre trained (model_type=model_type)
File "/yq/dmz/IntrutBLIP/LAVIS main/lavis/models/base_model. py", line 70, in from_ Pre trained
Model=cls. from_ Config (model_cfg)
File "/yq/dmz/InstrumentBLIP/LAVIS main/lavis/models/blip2_models/blip2_vicuna_instruct. py", line 729, in from_ Config
Model. load_ Checkpoint_ From_ Config (cfg)
File "/yq/dmz/IntrutBLIP/LAVIS main/lavis/models/base_model. py", line 102, in load_ Checkpoint_ From_ Config
Self load_ From_ Pre trained (url_or_filename=pre trained path, * * kwargs)
File "/yq/dmz/IntrutBLIP/LAVIS main/lavis/models/blip2_models/blip2. py", line 87, in load_ From_ Pre trained
Cached_ File=download_ Cached_ File（
File "/yq/dmz/IntrutBLIP/LAVIS main/navis/common/dist_utils. py", line 132, in download_ Cached_ File
Timm_ Hub. download_ Cached_ File (URL, check_hash, progress)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/site packages/tim/models/hub. py", line 51, in download_ Cached_ File
Download_ URL_ To_ File (URL, cached_file, hash_prefix, progress=progress)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/site packages/torch/hub. py", line 611, in download_ URL_ To_ File
U=urlopen (req)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 222, in urlopen
Return opener.open (URL, data, timeout)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 525, in open
Response=self_ Open (req, data)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 542, in_ Open
Result=self_ Call_ Chain (self. handle_open, protocol, protocol+
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 502, in_ Call_ Chain
Result=func (* args)
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 1397, in https_ Open
Return self.do_ Open (http. client. HTTPSConnection, req,
File "/yq/software/anaconde3/envs/navis/lib/Python 3.8/urllib/request. py", line 1357, in do_ Open
Raise URLError (err)
Urllib. error. URLError:<urlopen error [Errno 104] Connection reset by peer>

May I ask how to solve it

jameswan commented 1 year ago

Can I confirm that the yaml file to edit is: blip2_instruct_vicuna7b.yaml and the line to change is llm_model: ?

these files: pytorch_model-00001-of-00002.bin pytorch_model-00002-of-00002.bin

are stored in this folder: C:\Users\james.cache\huggingface\hub\models--lmsys--vicuna-7b-v1.5\snapshots\de56c35b1763eaae20f4d60efd64af0a9091ebe5

so should I change: llm_model: C:\Users\james.cache\huggingface\hub\models--lmsys--vicuna-7b-v1.5\snapshots\de56c35b1763eaae20f4d60efd64af0a9091ebe5