search

rapidsai / crossfit

Metric calculation library
Apache License 2.0
2 stars 5 forks source link

test example scripts in ci #40

Closed edknv closed 9 months ago

edknv commented 9 months ago

Note: Currently, each partition is processed in the Tokenizer the entire partition each time, so this leads to memory errors if the partition is large. We probably need to implement a loader so we can batch the text inputs in Tokenizer.

github-actions[bot] commented 9 months ago

Benchmark comparison for 05accee (main) vs 4cfd24a0 (PR)


--------------------------------------------------------------------------------------------- benchmark: 2 tests --------------------------------------------------------------------------------------------
Name (time in s)                                  Min                 Max               Mean            StdDev             Median               IQR            Outliers     OPS            Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fiqa_all_MiniLM_L6_v2 (0001_main)        95.0413 (1.0)      101.0205 (1.01)     98.2786 (1.0)      2.3209 (3.67)     98.3275 (1.0)      3.4636 (5.92)          2;0  0.0102 (1.0)           5           1
test_fiqa_all_MiniLM_L6_v2 (0002_4cfd24a)     98.6438 (1.04)     100.3055 (1.0)      99.7141 (1.01)     0.6332 (1.0)      99.8083 (1.02)     0.5852 (1.0)           1;0  0.0100 (0.99)          5           1
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
github-actions[bot] commented 9 months ago

Benchmark comparison for 05accee (main) vs 838c7017 (PR)


---------------------------------------------------------------------------------------------- benchmark: 2 tests ---------------------------------------------------------------------------------------------
Name (time in s)                                  Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fiqa_all_MiniLM_L6_v2 (0001_main)        96.7071 (1.0)       99.7920 (1.0)       98.7337 (1.0)      1.3241 (2.01)      99.3154 (1.0)      2.0141 (1.72)          1;0  0.0101 (1.0)           5           1
test_fiqa_all_MiniLM_L6_v2 (0002_838c701)     99.7444 (1.03)     101.2553 (1.01)     100.4115 (1.02)     0.6588 (1.0)      100.4380 (1.01)     1.1700 (1.0)           2;0  0.0100 (0.98)          5           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
edknv commented 9 months ago

Note: Currently, each partition is processed in the Tokenizer the entire partition each time, so this leads to memory errors if the partition is large. We probably need to implement a loader so we can batch the text inputs in Tokenizer.