How to make sure that only performing one step forward pass in profiling phase?

LiyuanLucasLiu / Transformer-Clinic

Understanding the Difficulty of Training Transformers

https://arxiv.org/abs/2004.08249

Apache License 2.0

326 stars 20 forks source link

How to make sure that only performing one step forward pass in profiling phase? #8

Closed ZhenYangIACAS closed 4 years ago

ZhenYangIACAS commented 4 years ago

Hello, I want to figure out how do you make sure that only performing one step forward pass in profiling phase in your code? Since the running scripts for profiling and training are the same, I cannot find where do you set the limitation for profiling.

LiyuanLucasLiu commented 4 years ago

Hi thanks for asking: -) Their scripts are not the same (--init-type adaptive is changed to --init-type adaptive-profiling for profiling).
The limitation is set at: https://github.com/LiyuanLucasLiu/Transformer-Clinic/blob/a302132c52084d730aca5822abb58228a04b6bb8/fairseq/fairseq/tasks/fairseq_task.py#L265