Closed qihqi closed 3 months ago
For Gemma 2B we need to change the shardings because the dimension we usually shard, num_kv_heads happens to be 1 for Gemma 2B. So we pick a different one to shard.
num_kv_heads
For Gemma 2B we need to change the shardings because the dimension we usually shard,
num_kv_heads
happens to be 1 for Gemma 2B. So we pick a different one to shard.