unable to load model

[segalinc]

Hello, I am trying to load the model using the demo but I get

    tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit=False, load_4bit=False, device="cuda")
  File "mPLUG-Owl/mPLUG-Owl2/mplug_owl2/model/builder.py", line 110, in load_pretrained_model
    vision_tower.to(device=device, dtype=torch.float16)
  File "venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1900, in to
    return super().to(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

[LianghuiGuo]

Same problem! I met this when using web ui

[LianghuiGuo]

Hello, I am trying to load the model using the demo but I get

    tokenizer, model, image_processor, context_len = load_pretrained_model(model_path, None, model_name, load_8bit=False, load_4bit=False, device="cuda")
  File "mPLUG-Owl/mPLUG-Owl2/mplug_owl2/model/builder.py", line 110, in load_pretrained_model
    vision_tower.to(device=device, dtype=torch.float16)
  File "venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1900, in to
    return super().to(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

Did you solve it? Thanks

[Tianchong-Jiang]

I solved this problem by using a GPU with larger RAM. This thread suggest that "This error is caused by Accelerate auto-offloading weights to either the CPU or disk because of insufficient memory on the GPU" so having sufficient GPU RAM solves it.