Document precision and determinism guarantees in Ginkgo

[upsj]

Motivated by #741:

Our SpMV formats and BLAS operations have different properties in terms of bitwise equivalence (determinism) and precision in general. It would make sense to document them more precisely.

Determinism

Important cases:

• Dense pointwise BLAS (axpy, ...) should be deterministic on a fixed executor/device
• Dense BLAS reductions (dot, norm, ...) are only deterministic when using the unified kernel reduction, AFAIK cuBLAS/hipBLAS make no such promises
• Dense GEMM is probably deterministic?
• COO is non-deterministic on all device executors due to the use of atomics
• CSR is non-deterministic on all GPU executors with load_balance and maybe sparselib (atomics), but deterministic with classical and merge_path
• ELL is non-deterministic on all GPU executors if using atomics
• SELL-P is deterministic on all executors

(Mixed) Precision

Dependent on whether the kernels use atomics or not, they can use the highest possible precision for all operations, or accumulate partially in the output precision. This has some impact on the precision on the precision of SpMV if the precision of the output vector(s) is smaller than the highest precision involved.

[upsj]

Examples where bitwise equivalence is important:

• MPI runs where all nodes need to do the exactly same calculations
• Debugging numerical issues is harder if non-determinism is involved

[MarcelKoch]

Just an organization question, do you suggest having a dedicated document somewhere which explains the all the details, or is this something that should be added to all classes/functions, where this might apply?

[upsj]

I would suggest adding a section to each LinOp (if it concerns apply) and function (if it concerns something else), and maybe a table giving an overview?