For Float16, why are the ouputs little different from Maple Diffusion?

[ghost]

I input the same exact tensors to Unet of both swift diffusin and maple diffuion, but outputs are little diffrent. The diffrence for each element is from 0.01 to 0.06.

Why is that so? I am using the NWHC branch

[ghost]

For FP32 they aint different

[liuliu]

For model compatibility reasons, we perform normalization operation at fp32: https://github.com/liuliu/ccv/blob/unstable/lib/nnc/cmd/norm/mps/ccv_nnc_layer_norm_mps.m#L63 I think maple-diffusion just do it at fp16: https://github.com/madebyollin/maple-diffusion/blob/main/maple-diffusion/MapleDiffusion.swift#L153

[ghost]

What compatibility reasons?

Thanks, is there any other differnce, apart from groupnorm and layernorm?

[ghost]

Right now, in swift-diffusion the FP16 output is very diffrent from swift-diffusion FP32 output. Although Maple's FP16 output is very close to Swift-diffusions FP32 output.

That is causing quality drop in swift diffusion for fp16.

[ghost]

There is some other difference too. i did layernorm and groupnorm at all fp16, still i am seeing a big difference.

[liuliu]

Thanks!

Try

DynamicGraph.flags = [.disableMixedMPSGEMM, .disableMixedMPSSoftMax]

See if you can get anywhere closer. If not, there might be something more.

[ghost]

Thanks a lot, it gets much closer. Btw this will fix the inpainting issue you have ( mentioned in the readme) .

now normal inpainting should work.

[ghost]

https://gist.github.com/brappier/c4101dcf0237e5025c94e53d873d9e05

[ghost]

What does MixedMPSGEMM and MixedMPSSoftMax do? Is there a bug in it?

[liuliu]

https://github.com/liuliu/ccv/blob/unstable/lib/nnc/cmd/softmax/mps/ccv_nnc_softmax_mps.m#L18

https://github.com/liuliu/ccv/blob/unstable/lib/nnc/cmd/blas/mps/ccv_nnc_gemm_mps.m#L123

You can dig deeper to see if particular one caused issues.

[liuliu]

Also, are you doing this with SD v1.x models or SD v2.x models?

[ghost]

1.x

[liuliu]

v2.x requires to disable MixedMPSGEMM but v1.x I don't find a reason to (I also didn't do numerical accuracy analysis like you did, just eyeballing end-result).