Detailed specification of the computer hardware to run 236B DeepSeek-Coder-V2

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

Apache License 2.0

734 stars 37 forks source link

Detailed specification of the computer hardware to run 236B DeepSeek-Coder-V2 #108

Open atomlayer opened 1 week ago

atomlayer commented 1 week ago

Local 236B DeepSeek-Coder-V2: Running its Q4_K_M version using only 21GB VRAM and 136GB DRAM

Can you provide more detailed specifications of the computer hardware? What kind of processor is needed, how many channels should RAM have? What type of RAM was used? DDR5?

I want to try to get this up and running, but I'm afraid I'll spend money on computer hardware that I won't be able to do anything with.

bitbottrap commented 5 days ago

Hold off. There are some issues that would need to be addressed before I'd spend money on a computer specifically dedicated to this software. The default configuration works well but its limited output would not be ideal for the common coding use case for this model. Efforts to increase output by changing some configuration values in code have encountered crashes and other limitations. It looks like it'd be really great once running.