search

rapidsai / crossfit

Metric calculation library
Apache License 2.0
2 stars 5 forks source link

add ops for running a custom pytorch classifier #38

Closed edknv closed 8 months ago

edknv commented 9 months ago

This PR moves most of the logic in Embedder to its new parent class Predictor. Predictor accepts any pytroch model and returns the outputs as a cudf dataframe. Embedder is a special case where we want the sentence_embedding field in the output dictionary, and hence inherits from Predictor.

The custom model in the sample script is copied almost verbatim from https://gitlab-master.nvidia.com/dl/JoC/nemo-data-curator/-/merge_requests/13 with a few minor modifications.

github-actions[bot] commented 9 months ago

Benchmark comparison for 2a7eefe (main) vs a1b8a319 (PR)


--------------------------------------------------------------------------------------------- benchmark: 2 tests ---------------------------------------------------------------------------------------------
Name (time in s)                                  Min                 Max                Mean            StdDev             Median               IQR            Outliers     OPS            Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fiqa_all_MiniLM_L6_v2 (0001_main)        97.6566 (1.0)      102.1067 (1.0)       99.6314 (1.0)      1.9383 (1.43)     99.9509 (1.00)     3.3780 (1.85)          2;0  0.0100 (1.0)           5           1
test_fiqa_all_MiniLM_L6_v2 (0002_a1b8a31)     98.8487 (1.01)     102.2326 (1.00)     100.0698 (1.00)     1.3551 (1.0)      99.6398 (1.0)      1.8300 (1.0)           1;0  0.0100 (1.00)          5           1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
github-actions[bot] commented 9 months ago

Benchmark comparison for 2a7eefe (main) vs deb571a3 (PR)


---------------------------------------------------------------------------------------------- benchmark: 2 tests ---------------------------------------------------------------------------------------------
Name (time in s)                                  Min                 Max                Mean            StdDev              Median               IQR            Outliers     OPS            Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_fiqa_all_MiniLM_L6_v2 (0001_main)        97.1364 (1.0)       99.9895 (1.0)       98.4655 (1.0)      1.2675 (1.0)       97.8458 (1.0)      2.1610 (1.0)           2;0  0.0102 (1.0)           5           1
test_fiqa_all_MiniLM_L6_v2 (0002_deb571a)     99.0572 (1.02)     102.5828 (1.03)     100.8467 (1.02)     1.7278 (1.36)     101.0998 (1.03)     3.4006 (1.57)          3;0  0.0099 (0.98)          5           1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean