search

pytorch / torchtitan

A native PyTorch Library for large model training
BSD 3-Clause "New" or "Revised" License
1.28k stars 115 forks source link

Remove PP+TP rmsnorm workaround #361

Closed wconstab closed 2 weeks ago

wconstab commented 1 month ago

Stack from ghstack (oldest at bottom):

fused_rmsnorm works with TP now, that was an old limitation no longer needed.

wconstab commented 2 weeks ago

i was wrong about this apparently.