Closed hiqsociety closed 11 months ago
can 8gb rtx 3060 run the 13b model?
I don't think so. In int4 quantization, the weights itself approach around 8G, and if taking KVCache into account, the upper bound could be 10-11 GB
can 8gb rtx 3060 run the 13b model?