expose use_orig_params to d2go config

facebookresearch / d2go

D2Go is a toolkit for efficient deep learning

Apache License 2.0

838 stars 201 forks source link

expose use_orig_params to d2go config #582

Closed YanjunChen329 closed 1 year ago

YanjunChen329 commented 1 year ago

Summary: Expose use_orig_params for FSDP constructor to d2go config. Read more about it in the docstring of torch.distributed.fsdp.fully_sharded_data_parallel.

use_orig_params=False (default) uses FlatParameters to store flattened parameters, which saves memory by avoiding fragmentation. However, use_orig_params=True is essential for models that are partly frozen. This is because FlatParameters can only accept uniform requries_grad across the whole model

Differential Revision: D46917757

facebook-github-bot commented 1 year ago

This pull request was exported from Phabricator. Differential Revision: D46917757

facebook-github-bot commented 1 year ago

This pull request has been merged in facebookresearch/d2go@7f17bbf02c03b59d9cd3a58fbc8ac74dd3fc7880.