BEYOND DATA AND MODEL PARALLELISM FOR DEEP NEURAL NETWORKS

[joapolarbear]

Introduction

• The execution simulator uses a delta simulation algorithm that simulates a new strategy using incremental updates to previous simulations.
• each oi’s configuration can be chosen independently from among all possible configurations for oi .
• Execution Simulator
• takes an operator graph G, a device topology D, and a parallelization strategy S as inputs and predicts the execution time
• measure the execution time of each distinct operator once for each configuration and include these measurements in a task graph
• only re-simulates tasks involved in the portion of the execution timeline that changes, an optimization that dramatically speeds up the simulator, especially for strategies for large distributed machines.