search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
475 stars 75 forks source link

TrOCR TT Implementation #1389

Closed boris-drazic closed 1 year ago

boris-drazic commented 1 year ago

Implement TrOCR using TT ops and TT fallback ops. (CPU implementation is in issue #1388.)

Make individual tests for significant submodules that are used in the model.

Each submodule test should use real weights from trained model and randomly generated inputs of correct shape.

Test should assert on PCC between golden output from PyTorch for submodule and TT output from module. Target PCC should be a parameter that pytest takes in.

vigneshkeerthivasanx commented 1 year ago

TT implementation for TrOCR Encoder which uses ViT model is completed. branch is here

vigneshkeerthivasanx commented 1 year ago

Currently working on TrOCR Decoder implementation.

vigneshkeerthivasanx commented 1 year ago

TT implementation for TrOCR Decoder is completed. PR is created for the same Link

vigneshkeerthivasanx commented 1 year ago

PR is updated with the changes Link

vigneshkeerthivasanx commented 1 year ago

Pushed patch for trocr_utils and test file (test_tt_trocr_causal_llm.py). Here is the commit The output from the CPU demo is "indus the" , Whereas the output from TtTrocr is "iwipps from his first Center to the state of New May-". This might be due to the low PCC of the decoder. Next token generated depends upon the previous token by the decoder. So one wrong token (First token) generated, triggers all the following tokens wrongly.

vigneshkeerthivasanx commented 1 year ago

PR is updated with the changes Link

saichandax commented 1 year ago

PR is merged. Closing the ticket.