Refactored benchmark tests

[shimizust]

Summary

• Refactored kernel benchmarking tests to save data to a single CSV (benchmark/data/all_benchmark_data.csv) that contains more complete information on how the benchmarking test was setup:

  kernel_name (e.g. swiglu)
  kernel_provider (e.g. liger or huggingface)
  kernel_operation_mode (e.g. full, forward, backward)
  metric_name (e.g. speed, memory)
  metric_unit (e.g. MB)
  x_name
  x_label (e.g. hidden size)
  x_value
  y_value_50 (median)
  y_value_20 (20th percentile)
  y_value_80 (80th percnetile)
  extra_benchmark_config_str (e.g. {""B"": 32, ""T"": 512, ""D"": 768, ""dtype"": ""torch.float32""})
  gpu_name (e.g. NVIDIA A100-SXM4-80GB)
  timestamp
  liger_version (e.g. 0.2.1)

• Removed existing image files to not clutter up the git history per previous discussion (https://github.com/linkedin/Liger-Kernel/pull/54)
• Added sample jupyter notebook that can be used to filter and plot the benchmark data
• Added makefile command to run all benchmarks: make run-benchmarks
• Don't overwrite existing data by default if same benchmark. Same benchmark means same kernel_name, kernel_provider, kernel_operation_mode, metric_name, x_label, gpu_name, and extra_benchmark_config_str

Testing Done

• Ran all benchmarks and spot-checked values with existing benchmark data

• Hardware Type: A100
• [x] run make test to ensure correctness
• [x] run make checkstyle to ensure code style
• [x] run make test-convergence to ensure convergence

[shimizust]

can we add a readme in benchmark/ to elaborate how to run these?

Added instructions to contributing.md

[shimizust]

Taking the raw results from triton.testing.do_bench here, we already get the unrounded values. I think the For example you can see in the CSV:

cross_entropy,liger,forward,speed,ms,V,vocab size,8192,0.8101439476013184,0.7565760016441345,0.9144319891929626,"{""B"": 8, ""T"": 2048}",NVIDIA A100-SXM4-80GB,2024-09-03 15:31:39,0.2.1

cc @ByronHsu @austin362667

[ByronHsu]

ah i see! you are writing the csv on your own