search

shikiw / OPERA

[CVPR 2024 Highlight] OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
MIT License
244 stars 22 forks source link

GPU information #21

Closed KlaineWei closed 4 months ago

KlaineWei commented 4 months ago

Hi, is there any detail information of the gpus used in the article?

shikiw commented 4 months ago

Hi,

All of our experiments use a single A100-80G GPU.

KlaineWei commented 4 months ago

Does the model support famous distributed training framework such as accelerate or deepseed?

shikiw commented 4 months ago

This codebase is just for evaluation but not for training. The key implementation of OPERA is in transformers-4.29.2/src/transformers/generation/utils.py and it is just a decoding algorithm for inference.

You can follow the Note in TL;DR to equip OPERA in other codebases or models that has accelerate or deepseed. Note that OPERA is for inference decoding, which is not relevant with these distributed training frameworks.

victorup commented 1 month ago

Hi @shikiw, how can I run the inference on two gpu devices?

Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)