152334H
commented
1 year ago - fp16 in the autoregressive model seems to degrade performance significantly; are there specific layers that can be forced to fp32 for better results?
- Less important, but quantisation could potentially be applied anywhere else in the model as well