Hi, when I ran ppo with bloomz-7b1-mt and bloom-560m (prompt_len = answer_len = 256) with zero stage 3 (8A100-40G), it seems the generation time is too slow (average about 72s). When I setting zero stage = 2, it consumes 0.8s with the same hyper-parameters for generation, but cause OOM after one step (I use totally 24A100-40G but still cause OOM). Currently, I consider the reason of generating slowly is using zero stage 3, but it consumes a lot of resource if I do not use this. Is there any solution for solving this problem? Thanks a lot!
Have you solved this problem? When I use PPO with zero 3 and cpu offload on the 2*8A800-80G to generate the length of 2000, it takes about 1500 seconds. Is this normal?
Hi, when I ran ppo with bloomz-7b1-mt and bloom-560m (prompt_len = answer_len = 256) with zero stage 3 (8A100-40G), it seems the generation time is too slow (average about 72s). When I setting zero stage = 2, it consumes 0.8s with the same hyper-parameters for generation, but cause OOM after one step (I use totally 24A100-40G but still cause OOM). Currently, I consider the reason of generating slowly is using zero stage 3, but it consumes a lot of resource if I do not use this. Is there any solution for solving this problem? Thanks a lot!