search

hpcaitech / ColossalAI-Examples

Examples of training models with hybrid parallelism using ColossalAI
Apache License 2.0
334 stars 102 forks source link

ZeRO 2 configuration example #71

Closed CHN-ChenYi closed 2 years ago

CHN-ChenYi commented 2 years ago

📚 The doc issue

The gpt2 zero2 example was removed after the new API was introduced. But I don't know how to not offload the model. I've tried removing model_config or setting offload_config to None/gpu/cuda, but neither of them works.

binmakeswell commented 2 years ago

Hi @CHN-ChenYi , we have update the example and tutorial, you can directly use zero, and do not need to set level 2.

https://github.com/hpcaitech/ColossalAI-Examples/tree/main/features/zero https://www.colossalai.org/docs/features/zero_redundancy_and_zero_offload