furiosa-ai / furiosa-models

FuriosaAI Model Zoo Project
https://furiosa-ai.github.io/furiosa-models/
Apache License 2.0
10 stars 6 forks source link

Adopt 0.9.1 changes to main #151

Closed furiosamg closed 1 year ago

furiosamg commented 1 year ago

0.9.1 버전을 급하게 릴리즈 하느라 branch-0.9.1 에서만 작업하고 관련 내용이 main 에 머지되지 않았었습니다. 그 외에, 우리 sdk 말고 외부 파이썬 패키지들의 변경 때문에 정확도 변경이 있었는데요, ci를 성공하기 위해 반영했습니다.

리뷰 받을 준비 완료되어 레디로 바꾸겠습니다! ㅎㅎ

furiosa-infra commented 1 year ago

42ca703 ci: add github-add-comment tkn command


----------------------------------------------------- benchmark: 1 tests ----------------------------------------------------
Name (time in ms)                       Min      Max    Mean  StdDev  Median     IQR   Outliers       OPS  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------
test_mlcommons_resnet50_accuracy     1.1273  82.6663  6.4307  6.0610  4.3001  1.6457  4745;7011  155.5042   50000           1
-----------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

42ca703 ci: add github-add-comment tkn command


-------------------------------------------------------------------------------------------------------- benchmark: 2 tests -------------------------------------------------------------------------------------------------------
Name (time in ms)                                                Min                   Max                Mean             StdDev              Median                IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mlcommons_ssd_resnet34_with_native_rust_pp_accuracy     40.6402 (1.0)        645.3738 (1.0)       61.8449 (1.0)      17.7657 (1.0)       61.2273 (1.0)       4.2201 (1.0)        40;248  16.1695 (1.0)        5000           1
test_mlcommons_ssd_resnet34_accuracy                         93.5940 (2.30)     1,203.0367 (1.86)     144.1386 (2.33)     33.0366 (1.86)     143.4377 (2.34)     20.0884 (4.76)      522;250   6.9378 (0.43)       5000           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

738d39d fix: update accuracies


----------------------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------------------
Name (time in ms)                                                Min                 Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mlcommons_ssd_mobilenet_with_native_rust_pp_accuracy     5.4056 (1.0)       23.3523 (1.0)      12.0214 (1.0)      3.0518 (1.0)      12.1962 (1.0)      4.8463 (1.27)       1899;1  83.1846 (1.0)        5000           1
test_mlcommons_ssd_mobilenet_accuracy                         7.2928 (1.35)     277.6437 (11.89)    17.1936 (1.43)     4.7074 (1.54)     17.5479 (1.44)     3.8182 (1.0)        552;42  58.1611 (0.70)       5000           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

738d39d fix: update accuracies


----------------------------------------------------- benchmark: 1 tests ----------------------------------------------------
Name (time in ms)                    Min         Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------
test_efficientnetv2s_accuracy     6.2004  1,031.9957  13.2879  7.5174  12.7080  2.8159  818;1620  75.2566   50000           1
-----------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

738d39d fix: update accuracies


---------------------------------------------------- benchmark: 1 tests ---------------------------------------------------
Name (time in ms)                   Min       Max     Mean  StdDev   Median     IQR   Outliers      OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------
test_efficientnetb0_accuracy     4.2670  438.7789  13.3492  8.0267  12.4042  4.1395  1165;1187  74.9111   50000           1
---------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

738d39d fix: update accuracies


------------------------------------------------------ benchmark: 1 tests -----------------------------------------------------
Name (time in ms)                       Min       Max    Mean   StdDev  Median     IQR   Outliers       OPS  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------
test_mlcommons_resnet50_accuracy     1.0971  143.3437  9.7180  13.3905  4.6725  1.8744  5838;8380  102.9018   50000           1
-------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosa-infra commented 1 year ago

6632ae3 fix: update yolov5l accuracy


----------------------------------------------------------------------------------------------------- benchmark: 2 tests ----------------------------------------------------------------------------------------------------
Name (time in ms)                                                Min                 Max               Mean            StdDev             Median               IQR            Outliers      OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mlcommons_ssd_mobilenet_with_native_rust_pp_accuracy     4.6429 (1.0)       25.1739 (1.0)      10.0829 (1.0)      3.5788 (1.00)      8.1501 (1.0)      6.9012 (3.88)       1784;1  99.1782 (1.0)        5000           1
test_mlcommons_ssd_mobilenet_accuracy                         8.4571 (1.82)     215.2643 (8.55)     18.3361 (1.82)     3.5628 (1.0)      18.5733 (2.28)     1.7785 (1.0)       453;463  54.5373 (0.55)       5000           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean