Detailed specification of the computer hardware to run 236B DeepSeek-Coder-V2

kvcache-ai / ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

Apache License 2.0

741 stars 39 forks source link

Detailed specification of the computer hardware to run 236B DeepSeek-Coder-V2 #108

Open atomlayer opened 3 weeks ago

atomlayer commented 3 weeks ago

Local 236B DeepSeek-Coder-V2: Running its Q4_K_M version using only 21GB VRAM and 136GB DRAM

Can you provide more detailed specifications of the computer hardware? What kind of processor is needed, how many channels should RAM have? What type of RAM was used? DDR5?

I want to try to get this up and running, but I'm afraid I'll spend money on computer hardware that I won't be able to do anything with.

bitbottrap commented 2 weeks ago

Hold off. There are some issues that would need to be addressed before I'd spend money on a computer specifically dedicated to this software. The default configuration works well but its limited output would not be ideal for the common coding use case for this model. Efforts to increase output by changing some configuration values in code have encountered crashes and other limitations. It looks like it'd be really great once running.