search

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
460 stars 69 forks source link

[On Hold] tr-ocr model gs demo Benchmarking #4486

Open Sudharsan-V opened 10 months ago

Sudharsan-V commented 10 months ago

GS e2e Demo

test uses batched real inputs and produces batched outputs

this test is looped (takes several batches of inputs, produces several batches of outputs)

test evaluates results of TT model against validation data in selected data set

test is documented

how to run test (commands to execute) what does the model do (high-level explanation in a couple of sentences) description of what is input and what is output this demo should load inputs and expected outputs from a data set on Weka (TBD which one)

jayasuryamaganuru commented 10 months ago

Benchmarking of tr-ocr model is done , Dataset: ICDAR_2003 Robust Word Recognition Sample set, downloaded from official website link Dataset is moved to WEKA path located at /mnt/MLPerf/tt_dnn-models/trocr/ICDAR_2003 Metric: Character error rate (CER)

CER : 4.0

Note: trocr model getting low PCC, thereby affecting its accuracy during benchmarking.

commit link

jayasuryamaganuru commented 9 months ago

On rebasing to latest main, the test hangs.

Able to reproduce the same results on main branch's older commit