junxnone / xwiki

https://junxnone.github.io/xwiki/

0 stars 0 forks source link

Hardware Precision #281

Open junxnone opened 1 year ago

junxnone commented 1 year ago

Precision

FP32
FP16
BF16
FP64
FP80
TF32

FP32

float
tf.float32
torch.float32/torch.float

TF32

tensor float 32
CUDA 用于提升 FLOPS

FP16

short float
tf.float16
torch.float16 / torch.half

BF16

tf.bfloat16
torch.bfloat16

bf16 vs fp16 vs fp32

FP64

double
tf.float64
torch.float64 / torch.double

FP80

long double