[🌟Tutorials🌟] Use MiniGPT-4 in Google Colab or your computer | 在本地或者Colab上使用MiniGPT-4

[WangRongsheng]

Use MiniGPT-4 in Colab

If you want to use miniGPT-4 in Google Colab, You must use GPU and you are a Google Colab Pro user, Otherwise you will not be able to use colab!

1. I provided a Minigpt-4 weight based on PrepareVicuna.md .
2. I provided a code in .

Use MiniGPT-4 in your computer

1. clone repo:

   https://github.com/Vision-CAIR/MiniGPT-4.git

2. install package:

   pip install -r requirements.txt

   requirements.txt are stored in WangRongsheng/Use-LLMs-in-Colab .

3. set config

   Set llama_model: "wangrongsheng/MiniGPT-4-LLaMA" in minigpt4/configs/models/minigpt4.yaml
   Set ckpt: 'pretrained_minigpt4.pth' in eval_configs/minigpt4_eval.yaml

4. run minigpt-4

   python demo.py --cfg-path eval_configs/minigpt4_eval.yaml 

Have good fun!

[TsuTikgiau]

wow, you are the best!

[WangRongsheng]

It is good!

[XuNing2]

运行!python demo.py --cfg-path eval_configs/minigpt4_eval.yaml出错

Initializing Chat
Downloading (…)solve/main/vocab.txt: 100% 232k/232k [00:00<00:00, 8.88MB/s]
Downloading (…)okenizer_config.json: 100% 28.0/28.0 [00:00<00:00, 4.18kB/s]
Downloading (…)lve/main/config.json: 100% 570/570 [00:00<00:00, 225kB/s]
Loading VIT
100% 1.89G/1.89G [00:11<00:00, 182MB/s]
Loading VIT Done
Loading Q-Former
100% 413M/413M [00:02<00:00, 187MB/s]
Loading Q-Former Done
Loading LLAMA
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /content/MiniGPT-4/demo.py:60 in <module>                                    │
│                                                                              │
│    57 model_config = cfg.model_cfg                                           │
│    58 model_config.device_8bit = args.gpu_id                                 │
│    59 model_cls = registry.get_model_class(model_config.arch)                │
│ ❱  60 model = model_cls.from_config(model_config).to('cuda:{}'.format(args.g │
│    61                                                                        │
│    62 vis_processor_cfg = cfg.datasets_cfg.cc_sbu_align.vis_processor.train  │
│    63 vis_processor = registry.get_processor_class(vis_processor_cfg.name).f │
│                                                                              │
│ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:243 in from_config           │
│                                                                              │
│   240 │   │   max_txt_len = cfg.get("max_txt_len", 32)                       │
│   241 │   │   end_sym = cfg.get("end_sym", '\n')                             │
│   242 │   │                                                                  │
│ ❱ 243 │   │   model = cls(                                                   │
│   244 │   │   │   vit_model=vit_model,                                       │
│   245 │   │   │   q_former_model=q_former_model,                             │
│   246 │   │   │   img_size=img_size,                                         │
│                                                                              │
│ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:86 in __init__               │
│                                                                              │
│    83 │   │   print('Loading Q-Former Done')                                 │
│    84 │   │                                                                  │
│    85 │   │   print('Loading LLAMA')                                         │
│ ❱  86 │   │   self.llama_tokenizer = LlamaTokenizer.from_pretrained(llama_mo │
│    87 │   │   self.llama_tokenizer.pad_token = self.llama_tokenizer.eos_toke │
│    88 │   │                                                                  │
│    89 │   │   if self.low_resource:                                          │
│                                                                              │
│ /usr/local/lib/python3.9/dist-packages/transformers/tokenization_utils_base. │
│ py:1771 in from_pretrained                                                   │
│                                                                              │
│   1768 │   │   │   │   elif is_remote_url(file_path):                        │
│   1769 │   │   │   │   │   resolved_vocab_files[file_id] = download_url(file │
│   1770 │   │   │   else:                                                     │
│ ❱ 1771 │   │   │   │   resolved_vocab_files[file_id] = cached_file(          │
│   1772 │   │   │   │   │   pretrained_model_name_or_path,                    │
│   1773 │   │   │   │   │   file_path,                                        │
│   1774 │   │   │   │   │   cache_dir=cache_dir,                              │
│                                                                              │
│ /usr/local/lib/python3.9/dist-packages/transformers/utils/hub.py:409 in      │
│ cached_file                                                                  │
│                                                                              │
│    406 │   user_agent = http_user_agent(user_agent)                          │
│    407 │   try:                                                              │
│    408 │   │   # Load from URL or cache if already cached                    │
│ ❱  409 │   │   resolved_file = hf_hub_download(                              │
│    410 │   │   │   path_or_repo_id,                                          │
│    411 │   │   │   filename,                                                 │
│    412 │   │   │   subfolder=None if len(subfolder) == 0 else subfolder,     │
│                                                                              │
│ /usr/local/lib/python3.9/dist-packages/huggingface_hub/utils/_validators.py: │
│ 112 in _inner_fn                                                             │
│                                                                              │
│   109 │   │   │   kwargs.items(),  # Kwargs values                           │
│   110 │   │   ):                                                             │
│   111 │   │   │   if arg_name in ["repo_id", "from_id", "to_id"]:            │
│ ❱ 112 │   │   │   │   validate_repo_id(arg_value)                            │
│   113 │   │   │                                                              │
│   114 │   │   │   elif arg_name == "token" and arg_value is not None:        │
│   115 │   │   │   │   has_token = True                                       │
│                                                                              │
│ /usr/local/lib/python3.9/dist-packages/huggingface_hub/utils/_validators.py: │
│ 160 in validate_repo_id                                                      │
│                                                                              │
│   157 │   │   raise HFValidationError(f"Repo id must be a string, not {type( │
│   158 │                                                                      │
│   159 │   if repo_id.count("/") > 1:                                         │
│ ❱ 160 │   │   raise HFValidationError(                                       │
│   161 │   │   │   "Repo id must be in the form 'repo_name' or 'namespace/rep │
│   162 │   │   │   f" '{repo_id}'. Use `repo_type` argument if needed."       │
│   163 │   │   )                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
HFValidationError: Repo id must be in the form 'repo_name' or 
'namespace/repo_name': '/path/to/vicuna/weights/'. Use `repo_type` argument if 
needed.

[WangRongsheng]

@XuNing2 Set llama_model: "wangrongsheng/MiniGPT-4-LLaMA" in minigpt4/configs/models/minigpt4.yaml

[sanjikk]

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

Loading checkpoint shards: 0% 0/3 [00:00<?, ?it/s] 卡在这里就结束了...

[WangRongsheng]

@sanjikk If you want to use miniGPT-4 in Google Colab, You must use GPU and you are a Google Colab Pro user, Otherwise you will not be able to use colab!

[sanjikk]

@WangRongsheng In fact, I am Pro user and use GPU.. Finally I find that I should choose High level GPU. Thanks

[ChristianAchenbach4815]

I am a Pro user and I have used the A100. But I get a "UnpicklingError: invalid load key, '<'."

Prompt Example

Human: Could you describe the contents of this image for me? ###Assistant:

Load BLIP2-LLM Checkpoint: pretrained_minigpt4.pth ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /content/MiniGPT-4/demo.py:60 in │ │ │ │ 57 model_config = cfg.model_cfg │ │ 58 model_config.device_8bit = args.gpu_id │ │ 59 model_cls = registry.get_model_class(model_config.arch) │ │ ❱ 60 model = model_cls.from_config(model_config).to('cuda:{}'.format(args.g │ │ 61 │ │ 62 vis_processor_cfg = cfg.datasets_cfg.cc_sbu_align.vis_processor.train │ │ 63 vis_processor = registry.get_processor_class(vis_processor_cfg.name).f │ │ │ │ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:265 in from_config │ │ │ │ 262 │ │ ckpt_path = cfg.get("ckpt", "") # load weights of MiniGPT-4 │ │ 263 │ │ if ckpt_path: │ │ 264 │ │ │ print("Load BLIP2-LLM Checkpoint: {}".format(ckpt_path)) │ │ ❱ 265 │ │ │ ckpt = torch.load(ckpt_path, map_location="cpu") │ │ 266 │ │ │ msg = model.load_state_dict(ckpt['model'], strict=False) │ │ 267 │ │ │ │ 268 │ │ return model │ │ │ │ /usr/local/lib/python3.9/dist-packages/torch/serialization.py:815 in load │ │ │ │ 812 │ │ │ │ return _legacy_load(opened_file, map_location, _weigh │ │ 813 │ │ │ except RuntimeError as e: │ │ 814 │ │ │ │ raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) │ │ ❱ 815 │ │ return _legacy_load(opened_file, map_location, pickle_module, │ │ 816 │ │ 817 │ │ 818 # Register pickling support for layout instances such as │ │ │ │ /usr/local/lib/python3.9/dist-packages/torch/serialization.py:1033 in │ │ _legacy_load │ │ │ │ 1030 │ │ │ f"Received object of type \"{type(f)}\". Please update to │ │ 1031 │ │ │ "functionality.") │ │ 1032 │ │ │ ❱ 1033 │ magic_number = pickle_module.load(f, pickle_load_args) │ │ 1034 │ if magic_number != MAGIC_NUMBER: │ │ 1035 │ │ raise RuntimeError("Invalid magic number; corrupt file?") │ │ 1036 │ protocol_version = pickle_module.load(f, pickle_load_args) │ ╰──────────────────────────────────────────────────────────────────────────────╯ UnpicklingError: invalid load key, '<'.

[WangRongsheng]

@ChristianAchenbach4815 Pleck check:

1. Set llama_model: "wangrongsheng/MiniGPT-4-LLaMA" in minigpt4/configs/models/minigpt4.yaml
2. Set ckpt: 'pretrained_minigpt4.pth' in eval_configs/minigpt4_eval.yaml

[WangRongsheng]

@TsuTikgiau Hi, I update MiniGPT-4 7B in Google Colab notebook, you can enjoy it!

[created-Bi]

hi, after setting llama_model: "wangrongsheng/MiniGPT-4-LLaMA" in minigpt4/configs/models/minigpt4.yaml, which llama model will it load? 13B or 7B

[WangRongsheng]

@created-Bi It will be help you: https://colab.research.google.com/drive/1OK4kYsZphwt5DXchKkzMBjYF6jnkqh4R?usp=sharing

[klocatelli]

@ChristianAchenbach4815 - The 13B model download URL is incorrect. The right URL is !wget https://huggingface.co/wangrongsheng/MiniGPT4/resolve/main/pretrained_minigpt4.pth (note "resolve/main" instead of "blob/main")

The "blob/main" URL is an HTML page, hence the error

!python demo.py --cfg-path eval_configs/minigpt4_eval.yaml --gpu-id 0

...
UnpicklingError: invalid load key, '<'

After this tiny change, I see no issue on Colab (running on A100) - Thanks @WangRongsheng 🥇

[created-Bi]

hello, when I running with wangrongsheng/MiniGPT-4-LLaMA-7B, an error happened that the shape of the weight and bias in the llama_proj module in original minigpt4 mismatched(4096 vs 5120). Thus, I'm wondering if you change the shape of the weight and bias in the llama_proj module?

[WangRongsheng]

@created-Bi Please give me more error information. I can't repeat this error.

[ArtemBernatskyy]

omg, this colab is GARBAGE, sorry but it so hard to use, don't commit half finished products

I know it's harsh but why on earth to use this colab we need to:

1. clone repo
2. edit repo to edit that files
3. edit colab to include our repo
4. we will receive error similar to @klocatelli (after this I just gave up and went here to shitpost)

[WangRongsheng]

@ArtemBernatskyy Here are some points to clarify:

1. There is no official release of minigpt-4 version, it is being improved, so you clone this repo in the colab.
2. You only need to edit two key parameters and some necessary environmental installation. If you don't want to do this, you can use the minigpt-4 demo
3. I don't understand what you are talking about.
4. Many people have run it in this way on both colab and local computers and they have all responded well. You should debug these errors, a perfect solution is not available, perhaps you can do better and look forward to your pull requestes.

[kuoyenlo]

@

@created-Bi Please give me more error information. I can't repeat this error.

Hi, I got the same error, here is the error information.

/usr/local/lib/python3.10/dist-packages/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported " 2023-05-01 08:46:15.855567: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Initializing Chat Loading VIT Loading VIT Done Loading Q-Former Loading Q-Former Done Loading LLAMA

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link

/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/lib/python3.10/dist-packages/cv2/../../lib64')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:105: UserWarning: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths... warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.28.0.1'), PosixPath('http'), PosixPath('8013')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-v100-hm-2nxtjzw2zpl6c --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true'), PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')} warn( CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.0 CUDA SETUP: Detected CUDA version 118 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so... Loading checkpoint shards: 100% 3/3 [02:17<00:00, 45.86s/it] Downloading (…)neration_config.json: 100% 137/137 [00:00<00:00, 96.5kB/s] Loading LLAMA Done Load 4 training prompts Prompt Example

Human: Describe this image in detail. ###Assistant:

Load BLIP2-LLM Checkpoint: /content/MiniGPT-4/prerained_minigpt4_7b.pth ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /content/MiniGPT-4/demo.py:60 in │ │ │ │ 57 model_config = cfg.model_cfg │ │ 58 model_config.device_8bit = args.gpu_id │ │ 59 model_cls = registry.get_model_class(model_config.arch) │ │ ❱ 60 model = model_cls.from_config(model_config).to('cuda:{}'.format(args.g │ │ 61 │ │ 62 vis_processor_cfg = cfg.datasets_cfg.cc_sbu_align.vis_processor.train │ │ 63 vis_processor = registry.get_processor_class(vis_processor_cfg.name).f │ │ │ │ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:266 in from_config │ │ │ │ 263 │ │ if ckpt_path: │ │ 264 │ │ │ print("Load BLIP2-LLM Checkpoint: {}".format(ckpt_path)) │ │ 265 │ │ │ ckpt = torch.load(ckpt_path, map_location="cpu") │ │ ❱ 266 │ │ │ msg = model.load_state_dict(ckpt['model'], strict=False) │ │ 267 │ │ │ │ 268 │ │ return model │ │ 269 │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:2041 in │ │ load_state_dict │ │ │ │ 2038 │ │ │ │ │ │ ', '.join('"{}"'.format(k) for k in missing_k │ │ 2039 │ │ │ │ 2040 │ │ if len(error_msgs) > 0: │ │ ❱ 2041 │ │ │ raise RuntimeError('Error(s) in loading state_dict for {} │ │ 2042 │ │ │ │ │ │ │ self.class.name, "\n\t".join(e │ │ 2043 │ │ return _IncompatibleKeys(missing_keys, unexpected_keys) │ │ 2044 │ ╰──────────────────────────────────────────────────────────────────────────────╯ RuntimeError: Error(s) in loading state_dict for MiniGPT4: size mismatch for llama_proj.weight: copying a param with shape torch.Size([4096, 768]) from checkpoint, the shape in current model is torch.Size([5120, 768]). size mismatch for llama_proj.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([5120]).

[kuoyenlo]

@WangRongsheng

Have you tried to run the finetune(stage 2) on colab? Such this command !torchrun --nproc-per-node 1 train.py --cfg-path train_configs/minigpt4_stage2_finetune.yaml

If I turn on the "low_resource: True" on the config minigpt4_stage2_finetune.yaml, The following issue about gpu and cpu happend>>>

/usr/local/lib/python3.10/dist-packages/requests/init.py:102: RequestsDependencyWarning: urllib3 (1.26.15) or chardet (5.1.0)/charset_normalizer (2.0.12) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported " 2023-05-01 12:30:28.549952: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT | distributed init (rank 0, world 1): env:// 2023-05-01 12:30:30,840 [INFO] ===== Running Parameters ===== 2023-05-01 12:30:30,841 [INFO] { "amp": true, "batch_size_eval": 6, "batch_size_train": 6, "device": "cuda", "dist_backend": "nccl", "dist_url": "env://", "distributed": true, "evaluate": false, "gpu": 0, "init_lr": 3e-05, "iters_per_epoch": 200, "lr_sched": "linear_warmup_cosine_lr", "max_epoch": 5, "min_lr": 1e-05, "num_workers": 2, "output_dir": "output/minigpt4_stage2_finetune", "rank": 0, "resume_ckpt_path": null, "seed": 42, "task": "image_text_pretrain", "train_splits": [ "train" ], "warmup_lr": 1e-06, "warmup_steps": 20, "weight_decay": 0.05, "world_size": 1 } 2023-05-01 12:30:30,841 [INFO] ====== Dataset Attributes ====== 2023-05-01 12:30:30,841 [INFO] ======== cc_sbu_align ======= 2023-05-01 12:30:30,841 [INFO] { "build_info": { "storage": "/content/cc_sbu_align/cc_sbu_align/" }, "data_type": "images", "text_processor": { "train": { "name": "blip_caption" } }, "vis_processor": { "train": { "image_size": 224, "name": "blip2_image_train" } } } 2023-05-01 12:30:30,841 [INFO] ====== Model Attributes ====== 2023-05-01 12:30:30,842 [INFO] { "arch": "mini_gpt4", "ckpt": "/content/MiniGPT-4/prerained_minigpt4_7b.pth", "drop_path_rate": 0, "end_sym": "###", "freeze_qformer": true, "freeze_vit": true, "image_size": 224, "llama_model": "wangrongsheng/MiniGPT-4-LLaMA-7B", "low_resource": true, "max_txt_len": 160, "model_type": "pretrain_vicuna", "num_query_token": 32, "prompt": "", "prompt_path": "prompts/alignment.txt", "prompt_template": "###Human: {} ###Assistant: ", "use_grad_checkpoint": false, "vit_precision": "fp16" } 2023-05-01 12:30:30,842 [INFO] Building datasets... Loading VIT 2023-05-01 12:30:57,246 [INFO] freeze vision encoder Loading VIT Done Loading Q-Former 2023-05-01 12:31:02,726 [INFO] load checkpoint from https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained_flant5xxl.pth 2023-05-01 12:31:02,733 [INFO] freeze Qformer Loading Q-Former Done Loading LLAMA

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link

/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/lib/python3.10/dist-packages/cv2/../../lib64')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:105: UserWarning: /usr/local/lib/python3.10/dist-packages/cv2/../../lib64:/usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths... warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8013'), PosixPath('//172.28.0.1'), PosixPath('http')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-v100-hm-2nxtjzw2zpl6c --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')} warn( /usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/tmp/torchelastic__pzc3ueu/nonex0s6uvw/attempt_0/0/error.json')} warn( CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so CUDA SETUP: Highest compute capability among GPUs detected: 7.0 CUDA SETUP: Detected CUDA version 118 CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so... Loading checkpoint shards: 100% 2/2 [01:11<00:00, 35.64s/it] Loading LLAMA Done Load 4 training prompts Prompt Example

Human: Describe this image in detail. ###Assistant:

Load BLIP2-LLM Checkpoint: /content/MiniGPT-4/prerained_minigpt4_7b.pth 2023-05-01 12:32:15,737 [INFO] Start training 2023-05-01 12:32:16,586 [INFO] dataset_ratios not specified, datasets will be concatenated (map-style datasets) or chained (webdataset.DataPipeline). 2023-05-01 12:32:16,586 [INFO] Loaded 3439 records for train split from the dataset. module.llama_proj.weight module.llama_proj.bias 2023-05-01 12:32:16,609 [INFO] number of trainable parameters: 3149824 2023-05-01 12:32:16,610 [INFO] Start training epoch 0, 200 iters per inner epoch. ╭───────────────────── Traceback (most recent call last) ──────────────────────╮ │ /content/MiniGPT-4/train.py:103 in │ │ │ │ 100 │ │ 101 │ │ 102 if name == "main": │ │ ❱ 103 │ main() │ │ 104 │ │ │ │ /content/MiniGPT-4/train.py:99 in main │ │ │ │ 96 │ runner = get_runner_class(cfg)( │ │ 97 │ │ cfg=cfg, job_id=job_id, task=task, model=model, datasets=datas │ │ 98 │ ) │ │ ❱ 99 │ runner.train() │ │ 100 │ │ 101 │ │ 102 if name == "main": │ │ │ │ /content/MiniGPT-4/minigpt4/runners/runner_base.py:378 in train │ │ │ │ 375 │ │ │ # training phase │ │ 376 │ │ │ if not self.evaluate_only: │ │ 377 │ │ │ │ logging.info("Start training") │ │ ❱ 378 │ │ │ │ train_stats = self.train_epoch(cur_epoch) │ │ 379 │ │ │ │ self.log_stats(split_name="train", stats=train_stats) │ │ 380 │ │ │ │ │ 381 │ │ │ # evaluation phase │ │ │ │ /content/MiniGPT-4/minigpt4/runners/runner_base.py:438 in train_epoch │ │ │ │ 435 │ │ # train │ │ 436 │ │ self.model.train() │ │ 437 │ │ │ │ ❱ 438 │ │ return self.task.train_epoch( │ │ 439 │ │ │ epoch=epoch, │ │ 440 │ │ │ model=self.model, │ │ 441 │ │ │ data_loader=self.train_loader, │ │ │ │ /content/MiniGPT-4/minigpt4/tasks/base_task.py:114 in train_epoch │ │ │ │ 111 │ │ log_freq=50, │ │ 112 │ │ accum_grad_iters=1, │ │ 113 │ ): │ │ ❱ 114 │ │ return self._train_inner_loop( │ │ 115 │ │ │ epoch=epoch, │ │ 116 │ │ │ iters_per_epoch=lr_scheduler.iters_per_epoch, │ │ 117 │ │ │ model=model, │ │ │ │ /content/MiniGPT-4/minigpt4/tasks/base_task.py:219 in _train_inner_loop │ │ │ │ 216 │ │ │ lr_scheduler.step(cur_epoch=inner_epoch, cur_step=i) │ │ 217 │ │ │ │ │ 218 │ │ │ with torch.cuda.amp.autocast(enabled=use_amp): │ │ ❱ 219 │ │ │ │ loss = self.train_step(model=model, samples=samples) │ │ 220 │ │ │ │ │ 221 │ │ │ # after_train_step() │ │ 222 │ │ │ if use_amp: │ │ │ │ /content/MiniGPT-4/minigpt4/tasks/base_task.py:68 in train_step │ │ │ │ 65 │ │ return datasets │ │ 66 │ │ │ 67 │ def train_step(self, model, samples): │ │ ❱ 68 │ │ loss = model(samples)["loss"] │ │ 69 │ │ return loss │ │ 70 │ │ │ 71 │ def valid_step(self, model, samples): │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in │ │ _call_impl │ │ │ │ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │ │ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │ │ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │ │ ❱ 1501 │ │ │ return forward_call(*args, kwargs) │ │ 1502 │ │ # Do not call functions when jit is used │ │ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │ │ 1504 │ │ backward_pre_hooks = [] │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py:115 │ │ 6 in forward │ │ │ │ 1153 │ │ │ │ │ is_joined_rank=False │ │ 1154 │ │ │ │ ) │ │ 1155 │ │ │ │ │ ❱ 1156 │ │ │ output = self._run_ddp_forward(*inputs, *kwargs) │ │ 1157 │ │ │ │ │ 1158 │ │ │ # sync params according to location (before/after forward │ │ 1159 │ │ │ # specified as part of hook, if hook was specified. │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/nn/parallel/distributed.py:111 │ │ 0 in _run_ddp_forward │ │ │ │ 1107 │ │ │ │ self.use_side_stream_for_tensor_copies, │ │ 1108 │ │ │ ) │ │ 1109 │ │ │ with self._inside_ddp_forward(): │ │ ❱ 1110 │ │ │ │ return module_to_run(inputs[0], kwargs[0]) # type │ │ 1111 │ │ else: │ │ 1112 │ │ │ with self._inside_ddp_forward(): │ │ 1113 │ │ │ │ return module_to_run(*inputs, kwargs) │ │ │ │ /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1501 in │ │ _call_impl │ │ │ │ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or s │ │ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hoo │ │ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks │ │ ❱ 1501 │ │ │ return forward_call(*args, *kwargs) │ │ 1502 │ │ # Do not call functions when jit is used │ │ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │ │ 1504 │ │ backward_pre_hooks = [] │ │ │ │ /content/MiniGPT-4/minigpt4/models/mini_gpt4.py:209 in forward │ │ │ │ 206 │ │ │ │ 207 │ │ to_regress_embeds = self.llama_model.model.embed_tokens(to_reg │ │ 208 │ │ inputs_embeds = torch.cat([bos_embeds, img_embeds, toregress │ │ ❱ 209 │ │ attention_mask = torch.cat([atts_bos, atts_img, to_regress_tok │ │ 210 │ │ │ │ 211 │ │ with self.maybe_autocast(): │ │ 212 │ │ │ outputs = self.llama_model( │ ╰──────────────────────────────────────────────────────────────────────────────╯ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument tensors in method wrapper_CUDA_cat) ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 75154) of binary: /usr/bin/python3 Traceback (most recent call last): File "/usr/local/bin/torchrun", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper return f(args, kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 794, in main run(args) File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 785, in run elastic_launch( File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train.py FAILED

Failures:

------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2023-05-01_12:32:56 host : f4e4433e4c5e rank : 0 (local_rank: 0) exitcode : 1 (pid: 75154) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html =======================================================

[ZhuJD-China]

why # Vicuna llama_model: "wangrongsheng/MiniGPT-4-LLaMA" , this model need to connect Internet When I run locally ??

[WangRongsheng]

why # Vicuna llama_model: "wangrongsheng/MiniGPT-4-LLaMA" , this model need to connect Internet When I run locally ??

Model weights are downloaded automatically, so you must be online.

[zhongpeixiang]

Thank you for the awesome work! One question, how can I change the download location for transformer models?

[ddholiday]

感谢精彩分享，已经跑起来了。另外，有了这个我是不是也可以部署一个单独的Vicuna？

[bakachan19]

can you share the code on how to use miniGPT4 on colab without gradio interface? Thank you!

[youyuanrsq]

can you share the code on how to use miniGPT4 on colab without gradio interface? Thank you! there are a demo you can try https://colab.research.google.com/drive/1VUzWoaGQoEx6OxgcRD742EbMpNlhAPHM?usp=sharing

[bakachan19]

Dear @youyuanrsq, Thank you!

After setting the llama_model and ckpt parameters it works!

• Set llama_model: "wangrongsheng/MiniGPT-4-LLaMA-7B" in minigpt4/configs/models/minigpt4.yaml
• Set ckpt: 'prerained_minigpt4_7b.pth' in eval_configs/minigpt4_eval.yaml

[autosquid]

@WangRongsheng git lfs pull权重的时候会报这个错

[Yuancheng-Xu]

can you share the code on how to use miniGPT4-V2 on colab without gradio interface? Thank you!