When HF Memory Metrics Disabled, the Benchmark CSV is Corrupted.

foundation-model-stack / fms-acceleration

🚀 Collection of libraries used with fms-hf-tuning to accelerate fine-tuning and training of large models.

Apache License 2.0

6 stars 12 forks source link

Closed fabianlim closed 1 month ago

fabianlim commented 3 months ago

When the HF memory metrics are disabled, running the benches as follows:

MEMORY_LOGGING=nvidia \
bash scripts/run_benchmarks.sh \
    ...

noticed that the benchmark.csv file will only have the framework_config and torch_dtype columns, the other columns will be empty,.

fabianlim commented 1 month ago

This has been addressed in #86, where we have added more entries into --keep_columns