WeiminLee
commented
4 months ago load model
model, model_args = AutoModel.from_pretrained(
args.from_pretrained,
args=argparse.Namespace(
deepspeed=None,
local_rank=0,
rank=0,
world_size=1,
model_parallel_size=1,
mode='inference',
skip_init=True,
use_gpu_initialization=True if torch.cuda.is_available() else False,
device='cuda',
overwrite_args={'model_parallel_size': 2},
**vars(args)
))
model.add_mixin('auto-regressive', CachedAutoregressiveMixin())
self.model = model.eval()
Remian777
ttq85
再请教一个问题，训练好的sat模型如何再两张GPU上加载并推理？ 我目前只有A100（40G）的版本，有时推理会报内u才能溢出。 请问如何设置多GPU推理？
下面模型加载部分如何设置呢？谢谢
# load model model, model_args = AutoModel.from_pretrained( args.from_pretrained, args=argparse.Namespace( deepspeed=None, local_rank=0, rank=0, world_size=1, model_parallel_size=1, mode='inference', skip_init=True, use_gpu_initialization=True if torch.cuda.is_available() else False, device='cuda', overwrite_args={'model_parallel_size': 2}, **vars(args) )) model.add_mixin('auto-regressive', CachedAutoregressiveMixin()) self.model = model.eval()