Camera matrix - Should the distortion params be incorporated?

[mnauf]

I am using a very simple camera matrix [[fx, 0, cx], [0, fy, cy], [0, 0, 1]]

Should any other distortion parameters be incorporated like tangential and radial distortion?

[xiexh20]

the distortion coefficients are stored in the intrinsic calibration files calibs/intrinsics/*/calibration.json. However, some differentiable renderer does not support these coefficients (e.g. pytorch3d). So I simply don't use them in rendering.