Open seanM29 opened 1 month ago
@tgale96 what are the performance implications for block size selection?
For now, to unblock I'd recommend forking or overriding the variable... but I'm not as sure here.
I recommend using the grouped code path rather than changing the block size. That is untested and likely to result in poor performance.
I use megablocks to implement a fine-granded moe, the ffn_hidden_size is divisible by 64, but is not divisible by 128, can we change it to 64? Thanks a lot