disable FSDP mixed precision for model buffers

facebookresearch / d2go

D2Go is a toolkit for efficient deep learning

Apache License 2.0

838 stars 201 forks source link

disable FSDP mixed precision for model buffers #585

Closed YanjunChen329 closed 1 year ago

YanjunChen329 commented 1 year ago

Summary: Disable FSDP mixed precision for model buffers. Buffers are usually small in size so there's very limited performance gain for enabling mixed precision. Plus, applications like BatchNorm layers and diffusion models are very sensitive to the precision of buffers. Thus, we stick to full precision for buffers in FSDP.

Differential Revision: D46951673

facebook-github-bot commented 1 year ago

This pull request was exported from Phabricator. Differential Revision: D46951673

facebook-github-bot commented 1 year ago

This pull request was exported from Phabricator. Differential Revision: D46951673

facebook-github-bot commented 1 year ago

This pull request has been merged in facebookresearch/d2go@b0abd7aa0024abb7c12c333db8bd3f48db054328.