Optimizing Large-scale Deep Learning by Minimizing Resource Contention for Data Processing

[joapolarbear]

Poster

• A good explanation of Horovod Workflow
• Solve the problem that horovod back ground threads needs to sync to each other to check which tensors are ready, which may be time consuming

Solution

1. Use global sleep time instead of local cycle time to avoid oversleep
2. Nonblocking Cache Synchronization
3. Static CPU Resource Partitioning
4. Graph Topology Exploitation, to ensure the tensor order.