Benchmarks for Metis

[YuMJie]

It is the excellent work "Metis: Fast Automatic Distributed Training on Heterogeneous GPUs", however, I have a couple of questions about the code:

1. Why are the configuration files execution_memory are same for different micro batch sizes?
2. Can you provide the profile files of different devices (e.g., RTX3090, etc.).
3. I see that the profile file configured in the README.md is about activation memory but this is not used in the code.

Could you provide the benchmarks for metis? Thank you!

[mgong-kang]

Thank you for your interest in this project.

1. As you mentioned, memory usage varies depending on the micro batch size. the files in profile_data_samples are based on a micro batch size of 1. It appears there was an error in copying the sample data, resulting in incorrect values. I will update this with the correct values.

2. While we do not have precise measurement data for the exact model provided in the sample, we will add some reference data that may be helpful. Additionally, you might consider regenerating and running the existing data to reflect approximate device performance as an alternative approach.

3. To measure pipeline communication costs more accurately, it is recommended to profile and use the activation size in advance. In the Metis code, a GPT model is provided, and it includes code for calculating the activation size of the GPT model, which has been used for the calculations.

[YuMJie]

Thank you for your reply: But I have some questions:

1. I have seen the code to calculate the activation size, but I can not find any code using the item "activation_parameters_bytes" in the profile file.
2. Could you provide the code for executing the config that Metis generates?
3. the item "parameters" in profiler file means the size of model, but using mixed-precision training, it exists two model weights which have different sizes, and which wight size should be writen?

Thank you!

[mgong-kang]

1. activation_parameter_bytes is not currently used in the code. It is a format prepared for models where calculating the activation size per model is challenging.
2. Are you referring to the code for generating the profile? If so, there is a generation guide in the README.md, and I kindly ask for your understanding as we cannot provide the code.
3. Since it is important for the communication cost to reflect the actual weight size of the model, it is deemed appropriate to use the FP32 weight size for measuring communication cost, even if some weights are converted to FP16 for computational efficiency in mixed-precision training.

Thank you!

[YuMJie]

I got it! Thank you for your reply! I will closed this issue

[YuMJie]

Hi, @mgong-kang I notice that you achieved the hete-aware data-parallelism load balanced, but I cannot see any code or output result about it. Could you provide some information? what is more. Could you provide how to use the metis config to run alpa? Thank you!

[mgong-kang]

The DataLoadBalancer is implemented at the follwing path: https://github.com/SamsungLabs/Metis/blob/main/model/load_balancer.py#L147

[mgong-kang]

@goeunee326 I would appreciate it if you could provide guidance on how to execute the Metis results in Alpa.

[YuMJie]

The DataLoadBalancer is implemented at the follwing path: https://github.com/SamsungLabs/Metis/blob/main/model/load_balancer.py#L147

Thank you for your help, but it seems that the output of the Metis strategy does not reflect the batch size when data parallelism in different GPUs.

What is more, I found some same strategies have different costs.

[mgong-kang]

@YuMJie

1. Data parallelism occurs when heterogeneous GPUs are allocated within a stage.

2. If you could send the profile data you've worked on, I'll take a look.

Thank you.