How to run Emu2-gen with native pytorch?

[cliangyu]

Hi! I tried to run Emu2-gen with native pytorch.

from emu.diffusion import EmuVisualGeneration
from PIL import Image
import numpy as np
import cv2
import torch

pipe = EmuVisualGeneration.from_pretrained(
        "/data/lychen/code/decoding/emu2/Emu2-Gen_pytorch_model.bf16.safetensors", # "/home/lychen/.cache/huggingface/hub to Emu2-Gen_pytorch_model.bf16.safetensors",
        dtype=torch.bfloat16,
        use_safetensors=True,
)

I got error

Traceback (most recent call last):
  File "/data/lychen/code/decoding/emu2/Emu/Emu2/gen.py", line 7, in <module>
    pipe = EmuVisualGeneration.from_pretrained(
  File "/data/lychen/code/decoding/emu2/Emu/Emu2/emu/diffusion.py", line 257, in from_pretrained
    return cls(
  File "/data/lychen/code/decoding/emu2/Emu/Emu2/emu/diffusion.py", line 39, in __init__
    print(self.emu_model.load_state_dict(encoder, strict=False))
  File "/data/lychen/code/decoding/emu2/Emu/Emu2/emu/emu.py", line 236, in load_state_dict
    state_dict = torch.load(state_dict, map_location="cpu")
  File "/home/lychen/.local/lib/python3.10/site-packages/torch/serialization.py", line 986, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/home/lychen/.local/lib/python3.10/site-packages/torch/serialization.py", line 435, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/lychen/.local/lib/python3.10/site-packages/torch/serialization.py", line 416, in __init__
    super().__init__(open(name, mode))
NotADirectoryError: [Errno 20] Not a directory: '/data/lychen/code/decoding/emu2/Emu2-Gen_pytorch_model.bf16.safetensors/multimodal_encoder/pytorch_model.bin'

[ryanzhangfan]

You may use the huggingface version of code to load native pytorch version model. Please make sure you are using EmuVisualGeneration in here.

[cliangyu]

Thank you! I pulled the latest code and resolved the issue.