search

chengchingwen / Transformers.jl

Julia Implementation of Transformer models
MIT License
523 stars 74 forks source link

addresses issue #92 #93

Closed Alexander-Barth closed 2 years ago

Alexander-Barth commented 2 years ago

This PR addresses issue #92.

By setting n temporarily to 100, I get the same values as show in [1]:

julia> Transformers.Basic.PE.(4,(1:4),(1:4)')                                                                                                                         
4×4 Matrix{Float64}:                                                                                                                                                  
  1.0       0.0       1.0       0.0                                                                                                                                   
  0.540302  0.841471  0.995004  0.0998334                                                                                                                             
 -0.416147  0.909297  0.980067  0.198669                                                                                                                              
 -0.989992  0.14112   0.955336  0.29552

(but this PR does not touches n and keeps its original value of n = 1e4 😀)

[1] https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/

codecov[bot] commented 2 years ago

Codecov Report

Merging #93 (cc3960b) into master (7a7a25a) will not change coverage. The diff coverage is 100.00%.

:exclamation: Current head cc3960b differs from pull request most recent head bc444fc. Consider uploading reports for the commit bc444fc to get more accurate results

@@           Coverage Diff           @@
##           master      #93   +/-   ##
=======================================
  Coverage   22.05%   22.05%           
=======================================
  Files          69       69           
  Lines        3378     3378           
=======================================
  Hits          745      745           
  Misses       2633     2633
Impacted Files Coverage Δ
src/basic/embeds/position_embed.jl 70.27% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7a7a25a...bc444fc. Read the comment docs.

chengchingwen commented 2 years ago

Thanks!