ML AutoML NAS ProxylessNAS

[junxnone]

ProxylessNAS

• Proxy Task -> Target Task
• Proxy - 小数据集(Cifar)/Fewer Blocks/Fewer Epoch -> 大数据 (ImageNet)/Stacked Top-Performing Bloocks/Retraining
• 把 NAS 视为 path-level pruning 的过程
• Path-Level pruning - prune redundant parts
• Binarized Architecture parameters - Binary Gate(0-prune/1-keep)
• DAG
  • 之前的 NAS 把 operation 作为 edge
  • Proxyless NAS 使用 Mixed Operation 作为一个 edge, 即 (operation + Binary Gate) x N

Proxy-Based vs Proxy-Less -- |

Arch

Binarized Architecture parameters -- |

Cifar-10 Op Set	ImageNet Op Set (various kernel sizes/expansion ratios)
3 × 3 dilated depthwise-separable convolution	MBConv layers 3x3/3
Identity	MBConv layers 3x3/6
3 × 3 depthwise-separable convolution	MBConv layers 5x5/3
5 × 5 depthwise-separable convolution	MBConv layers 5x5/6
7 × 7 depthwise-separable convolution	MBConv layers 7x7/3
3 × 3 average pooling	MBConv layers 7x7/6

3 × 3 max pooling

Name	DARTS	ProxyLess
Output	Operation Output 求和	Binary Gate 取一个路径 Output
计算	每次计算 N Path	每次计算 1 Path
Memory	M	M x N
Update Weights	standard gradient descent	BinaryConnect

Latency Differentiable

• 抽象成连续函数
• Latency regularization Loss

Search Result - GPU/CPU/Mobile

Reference

• paper - 2018.12 - ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Github
• 【ProxylessNAS】2019-ICLR-ProxylessNAS Direct Neural Architecture Search on Target Task and Hardware-论文阅读
• ProxyLessNAS

[junxnone]

junxnone/tech-io#805 junxnone/aiwiki#393