Open Luosuu opened 1 year ago
It looks a little confusing and maybe could be written more clearly, but I think that's actually correct. If it was just model
, it would be trying to FSDP-wrap the DDP model. By using model()
, it's FSDP wrapping a new model instance.
📚 Documentation
In the blog introducing FSDP API
it should be
model
instead ofmodel()
insideFullyShardedDataParallel
so it should be