Open Pattaro opened 10 months ago
me too... Did you solve this problem?
@Pattaro, this happens with parameter partitioning of zero stage 3. The parameters will be fetched on-demand before use, so no reason for alarm. Are you seeing any training issues otherwise?
****[start] Initializing Reward Model [start] **** [2023-11-29 14:57:02,054] [INFO] [partition_parameters.py:347:exit] finished initializing model - num_params = 1306, num_elems = 39.25B