Better Models for Stochastic Optimization

[JonathanChiang]

Robustness is Important:

• Capacity and Trainability in Recurrent Neural Networks:
• Energy Spent in Training???????
• Camry to SF to LA
• Robustness:
• common RNN architectures achieve the same per-task
• Camry in 10 -> 100 -> 1000 -> 10,000

  Stochastic Gradient Methods:

• minimize a function:

  Weakly Convex Functions:

  Why We use This??

• easy to analyze:
• default packages?
• works?

  Linear Regression:

• U- shaped algorithm (sgm, truncated, prox)

  Optimization Methods:

• good but simple local model of f
• minimize the model regularizing

  Optimization Methods:

• how to solve optimization problems:
• minimize a model (regularizing)

  Newton's Method:

• Taylor (second-order) model:

  Composite Optimization Problems:

  Modeling Composite Problems:

• convex model

  Modeling Composite Problems:

• now we make a convex model

  Modeling Composite Problems

  Generic Optimization Methods:

  aProx family for stochastic optimization:

  Models in Stochastic Optimization:

• conditions on our models (convex case)
• lower bound
• local correctness

  Divergence of a gradient method

  Conclusion:

• blind application of SGD is not right answer
• care and better modeling can yield improved performance
• computational efficieny c