ML AutoML Tools SIGOPT

[junxnone]

SIGOPT

• Online Tools - 跟踪/分析/微调 AI Model
• sigopt UseCase
• sigopt Options - parameters/metrics
• 参数类型支持 浮点型/整型/Grid/Categorical
• Multimetric (用例: 优化 accuracy 的同时, 优化 inference_time)
• Support Optimization: Random/Grid/Bayesian/Constraint/Custom
  • Custom Optimizer
  • Hyperopt
  • Optuna
• Parallel
  • 多机器训练

Arch

Workflow

• 1. 获取随机超参
• 2. 把跑出来的 metrics 提交到 sigopt
• 3. 获取新的超参
• 4. 重复以上步骤获取最好的参数

Parallel

• 方式
  • Threads
  • Machines (+unique tag)
  • Training clusters
• Pipelines
  • set parallel_bandwidth
  • Init workers with EXPERIMENT_ID
  • Optimization Loop - Suggestions/Training/Evaluation/Observations

Reference

• SigOpt docs [code]
• 2016 A Stratified Analysis of Bayesian Optimization Methods [paper]
• sigopt_experiment_and_optimization_demo.ipynb
• Intro to Multicriteria Optimization