Performance metrics

[stsievert]

• [ ] how effective are different compression? This will require passing some argument from train.py
• [ ] given the same number of bytes communicated, which coding is most effective? This will require looking at gradient variance increase (already coded)
  • Only do this for one network, all algorithms. Do not show over iterations (no tuning step)
• [x] See how rank changes with reshaping
  • Only do this for one network, all algorithms. Do not show over iterations (no tuning step).

[stsievert]

What the status on this @hwang595?