[MLSys'19] Beyond Data and Model Parallelism for Deep Neural Networks

[ganler]

https://mlsys.org/Conferences/2019/doc/2019/16.pdf

Yet another hardcore work from Zhihao.

[ganler]

Challenges

• Get a good parallel execution method in 4 dims.
• Speedup the profiling.

The SOAP search space

• Samples: data parallelism(e.g., samples within a batch)
• Operators: Model parallelism(place different operators across different devices)
• Attributes: elements within a sample. (like what people do in training big models)
• Parameters: parameters within an operator.

Cost

Measuring distributed execution on real hardware is slow.

2 Obs.

• The performance of DNN operator is highly predictable.
• DNN models only leverage a small set of distinct operators.

Execution simulator:

• Profile each distinct operator once.
• Use the measurements to different estimate different parallelization strategies.

Operator Graph to Task Graph.

Delta Simulation Algo.

Incremental update. (Less re-profiling)

Reduce search time by 2-7x => take only a few minutes.

What we need in a paper

• Explanations of the algorithms.
• Evaluation.
• Case study to uncover a deeper explanation.