Closed KlaineWei closed 4 months ago
Hi,
All of our experiments use a single A100-80G GPU.
Does the model support famous distributed training framework such as accelerate or deepseed?
This codebase is just for evaluation but not for training. The key implementation of OPERA is in transformers-4.29.2/src/transformers/generation/utils.py
and it is just a decoding algorithm for inference.
You can follow the Note in TL;DR to equip OPERA in other codebases or models that has accelerate or deepseed. Note that OPERA is for inference decoding, which is not relevant with these distributed training frameworks.
Hi @shikiw, how can I run the inference on two gpu devices?
Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
Hi, is there any detail information of the gpus used in the article?