Open certik opened 1 year ago
This approximates the erf() function directly.
To use it, apply the following patch:
--- a/gpt2.f90 +++ b/gpt2.f90 @@ -109,7 +109,7 @@ real(sp), intent(in) :: x(:,:), fc_w(:,:), fc_b(:), proj_w(:,:), proj_b(:) real(sp) :: y(size(x,1),size(x,2)) !real(sp) :: a(4*size(x,1),size(x,2)) !a = gelu(linear(x, fc_w, fc_b)) -y = linear(gelu(linear(x, fc_w, fc_b)), proj_w, proj_b) +y = linear(fast_gelu(linear(x, fc_w, fc_b)), proj_w, proj_b) end function function attention(n_embd_head,n_seq,n_seq_x, q, k, v, mask) result(y)
This approximates the erf() function directly.
To use it, apply the following patch: