I am following https://huggingface.co/docs/transformers/model_doc/bark#combining-optimizaton-techniques. I am on a single gpu ml.g5.16xlarge Linux machine on AWS Sagemekr (specs: https://aws.amazon.com/ec2/instance-types/g5/) . I have already installed the bark from its git repo, and have it working with no issues on this machine. The issue comes in when I try to implement this optimized form described, using transformers (instead of from bark import SAMPLE_RATE, generate_audio, preload_models). I have exactly copied and paste the code, and get the following error, when I do the audio_array = model.generate(**inputs)
my code:
from transformers import AutoProcessor, BarkModel
from IPython.display import Audio
import torch
from optimum.bettertransformer import BetterTransformer
device = "cuda" if torch.cuda.is_available() else "cpu"
# load in fp16
model = BarkModel.from_pretrained("suno/bark-small", torch_dtype=torch.float16).to(device)
# convert to bettertransformer
model = BetterTransformer.transform(model, keep_original_model=False)
# enable CPU offload
model.enable_cpu_offload()
processor = AutoProcessor.from_pretrained("suno/bark")
# model = BarkModel.from_pretrained("suno/bark")
voice_preset = "v2/en_speaker_6"
inputs = processor("Hi, my name is Frank", voice_preset=voice_preset)
audio_array = model.generate(**inputs)
audio_array = audio_array.cpu().numpy().squeeze()
the error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
I am following https://huggingface.co/docs/transformers/model_doc/bark#combining-optimizaton-techniques. I am on a single gpu ml.g5.16xlarge Linux machine on AWS Sagemekr (specs: https://aws.amazon.com/ec2/instance-types/g5/) . I have already installed the bark from its git repo, and have it working with no issues on this machine. The issue comes in when I try to implement this optimized form described, using transformers (instead of
from bark import SAMPLE_RATE, generate_audio, preload_models
). I have exactly copied and paste the code, and get the following error, when I do theaudio_array = model.generate(**inputs)
my code:
the error:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)
Traceback: