search

furiosa-ai / furiosa-models

FuriosaAI Model Zoo Project
https://furiosa-ai.github.io/furiosa-models/
Apache License 2.0
9 stars 6 forks source link

Update artifacts 0.9.0 #141

Closed sukyoungjeong-furiosa closed 1 year ago

sukyoungjeong-furiosa commented 1 year ago

NOTE: These artifacts are built on sdk 0.9.0-rc5.

Updated calibration methods on efficientnet-b0, with updated accuracy to 73.556 (previously, 16.104).

>       assert accuracy == EXPECTED_ACCURACY, "Accuracy check failed"
E       AssertionError: Accuracy check failed
E       assert 73.556 == 16.104

tests/bench/test_efficientnet_b0.py:56: AssertionError

---------------------------------------------------- benchmark: 1 tests ---------------------------------------------------
Name (time in ms)                   Min       Max     Mean  StdDev   Median     IQR   Outliers      OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------
test_efficientnetb0_accuracy     8.0916  457.8515  20.9568  8.9175  20.4500  5.3315  1463;1017  47.7171   50000           1
---------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
====================================================== short test summary info =======================================================
FAILED tests/bench/test_efficientnet_b0.py::test_efficientnetb0_accuracy - AssertionError: Accuracy check failed
=================================================== 1 failed in 1060.82s (0:17:40) ===================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████▉| 49999/50000 [17:33<00:00, 47.46it/s]

Also for efficientnetv2s, as follows. (75.208 => 83.566)

>       assert accuracy == EXPECTED_ACCURACY, "Accuracy check failed"
E       AssertionError: Accuracy check failed
E       assert 83.566 == 75.208

tests/bench/test_efficientnet_v2_s.py:56: AssertionError

------------------------------------------------------ benchmark: 1 tests ------------------------------------------------------
Name (time in ms)                     Min         Max     Mean   StdDev   Median     IQR   Outliers      OPS  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------
test_efficientnetv2s_accuracy     16.2072  1,161.0729  45.7795  11.6128  44.8509  3.7553  1488;3048  21.8439   50000           1
--------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
====================================================== short test summary info =======================================================
FAILED tests/bench/test_efficientnet_v2_s.py::test_efficientnetv2s_accuracy - AssertionError: Accuracy check failed
=================================================== 1 failed in 2303.66s (0:38:23) ===================================================
100%|██████████████████████████████████████████████████████████████████████████████████████████▉| 49999/50000 [38:20<00:00, 21.74it/s]