search

anarchy-ai / LLM-VM

irresponsible innovation. Try now at https://chat.dev/
https://anarchy.ai/
MIT License
465 stars 150 forks source link

Add support to change dtype for a model? #404

Closed MehmetMHY closed 6 months ago

MehmetMHY commented 7 months ago

I think a useful parameter/feature LLM-VM should support is the ability to change the dtype of a model. In this HuggingFace doc, it is mentioned that this can be done by doing something like this:

model.to(dtype=torch.bfloat16)

I am doing something like this in llm-speed-benchmark, so I do think this is possible.

I would love to get peoples' option on this as well.

mobley-trent commented 7 months ago

Hello @VictorOdede can I work on this issue ?

VictorOdede commented 7 months ago

@mobley-trent Go ahead!