search

BatsResearch / bonito

A lightweight library for generating synthetic instruction tuning datasets for your data without GPT.
BSD 3-Clause "New" or "Revised" License
697 stars 46 forks source link

How can I set the gpu_memory_utilization? #22

Closed Bearljj closed 6 months ago

Bearljj commented 6 months ago

I encountered this error when running the code:

ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.

But I have already made these settings

bonito = Bonito("BatsResearch/bonito-v1",gpu_memory_utilization=0.9)

, it seems like they are not working, what should I do?

nihalnayak commented 6 months ago

This is related to the issue in the vllm package (https://github.com/vllm-project/vllm/issues/2248).

You could try the following with Bonito and see if that helps: 

bonito = Bonito("BatsResearch/bonito-v1", enforce_eager=True)