[Doptim] refine APIs and add more checks for use_distributed_optimizer settings

volcengine / veScale

A PyTorch Native LLM Training Framework

http://vescale.xyz

Apache License 2.0

553 stars 26 forks source link

[Doptim] refine APIs and add more checks for use_distributed_optimizer settings #15

Closed Vremold closed 5 months ago

Vremold commented 5 months ago

As detailed in the titles, this PR refines the API of BasicOptimizer and add more checks for use_distributed_optimizer settings. In this PR, we also add a test case about shared weight on single machine. We will later release an integration test when our PP is ready.