It seems necessary to switch from directly executing the low-level nodes of the IR graph transformed with torch.fx to executing the high-level GraphModule processed through split_module(). The reason for this is to address the relatively slow training of GPT-like models in the low-level execution framework, even though the training of composite models is fast.
It seems necessary to switch from directly executing the low-level nodes of the IR graph transformed with torch.fx to executing the high-level GraphModule processed through split_module(). The reason for this is to address the relatively slow training of GPT-like models in the low-level execution framework, even though the training of composite models is fast.