search

amazon-science / RefChecker

RefChecker provides automatic checking pipeline and benchmark dataset for detecting fine-grained hallucinations generated by Large Language Models.
Apache License 2.0
226 stars 17 forks source link

Getting Cuda out of memory error #6

Closed isspek closed 5 months ago

isspek commented 5 months ago

Hi,

Thanks for the repository. I am running the code on a laptop, so the GPU memory is limited if I use open LLMs (e.g. extractor = MistralExtractor()). Is there a way to setup batch size while using these models?

HuXiangkun commented 5 months ago

Hi @isspek , the extractors are not using batching for the extraction. The Mistral extractor has 7B parameters, so it is not able to deployed into a laptop, so I think you need a GPU to run the Mistral-based extractor.

isspek commented 5 months ago

Thanks for the clarification @HuXiangkun