Some additional datapoints

[TortoiseHam]

I have some additional data points if you're interested:

M1 Max 32 Core (64GB)

torch-1.13.0.dev20220628-cp310-none-macosx_11_0_arm64.whl

• Training
  • Sequence Length 128
    • Batch Size 16: 43.20 seconds
    • Batch Size 64: 111.21 seconds
  • Sequence Length 512
    • Batch Size 16: 146.23 seconds
• Inference
  • Sequence Length 128
    • Batch Size 16: 9.24 seconds
    • Batch Size 64: 30.98 seconds
    • Batch Size 256: 118.24 seconds
  • Sequence Length 512
    • Batch Size 16: 36.00 seconds
    • Batch Size 64: 138.88 seconds
    • Batch Size 256: 560.17 seconds

Based on watching iStat while the training is running, it seems like larger training jobs aren't really spinning up the GPU much, nor heavily occupying the gpu memory. The jump from a few minutes up to hours of time might be a software related issue rather than necessarily a hardware limitation