How to get hands on experience as a newbie - Githubissues

databricks / dbrx

Code examples and resources for DBRX, a large language model developed by Databricks

https://www.databricks.com/

Other

2.47k stars 231 forks source link

How to get hands on experience as a newbie #24

Closed simkimsia closed 3 months ago

simkimsia commented 3 months ago

My first option is to run quantized versions.

Quantized

I read this https://github.com/databricks/dbrx#mlx

and then went to https://huggingface.co/mlx-community/dbrx-instruct-4bit

I read this

On my Macbook Pro M2 with 96GB of Unified Memory, DBRX Instruct in 4-bit for the above prompt it eats 70.2GB of RAM.

I am on a macbook pro M1 Max with 64Gb memory.

I guess that's not enough?

Computing

My next version is to figure out what's a cheap way to run the model but the details confuse me.

Can help?

simkimsia commented 3 months ago

sorry i shifted this to the community discussion on hugging face https://huggingface.co/databricks/dbrx-instruct/discussions/38