Why is the inference speed using fp16 or bf16 similar to fp32?

[lionday]

Is there any specific configuration method? model = AutoModelForCausalLM.from_pretrained(checkpoint,trust_remote_code=True,torch_dtype=torch.float16)

[loubnabnl]

can you give more details on how you're doing generation? fp16 and bf16 are it's faster in my experiments