search

chengchingwen / Transformers.jl

Julia Implementation of Transformer models
MIT License
523 stars 74 forks source link

Update docs and examples for onehot/vocabulary with Int input #66

Closed lazarusA closed 2 years ago

lazarusA commented 3 years ago

I'm testing some code from the docs. In this case the copy task.

https://chengchingwen.github.io/Transformers.jl/dev/tutorial/#define-the-loss-and-training-loop

I notice that things go wrong in the loss function definition. Due to the onehot method.

By running that code one gets this error. 

ERROR: LoadError: MethodError: no method matching OneHotArray(::Transformers.Basic.OneHot{0x0000000d})
Closest candidates are:
  OneHotArray(::Any, ::Transformers.Basic.OneHot)

this the env that I'm using. 

(transformersApproach) pkg> st
      Status `~/transformersApproach/Project.toml`
  [052768ef] CUDA v3.3.4
  [587475ba] Flux v0.12.5
  [21ca0261] Transformers v0.1.13
  [e88e6eb3] Zygote v0.6.17

any quick work around this issue?

chengchingwen commented 3 years ago

The quickest fix would be replace label = onehot(vocab, y) with label = Basic.OneHotArray(length(vocab), y). It's because we are using a rare example where the input sequence is of type Int in the tutorial. If you are going to modified the example to do some real task, than use the original one (onehot(vocab, y))

chengchingwen commented 3 years ago

And this happen because some API change in Transformers v0.1.13. I should update the docs/example

lazarusA commented 3 years ago

ok. This fixes the issue here. Thanks.