PaddlePaddle / models

Officially maintained, supported by PaddlePaddle, including CV, NLP, Speech, Rec, TS, big models and so on.
Apache License 2.0
6.91k stars 2.91k forks source link

paddle fleet分布式框架 设置参数不同学习率时报错 #5738

Open lwgkzl opened 1 year ago

lwgkzl commented 1 year ago

paddle版本: 2.2.1. 背景: 多任务学习中,同一个backbone模型有不同的头学习不同的任务, 期望各个头fc层的学习率可以不一样,但是设置的时候会有如下错误: 错误信息如下: Traceback (most recent call last): File "run_batch_fine_grained.py", line 466, in train(args) File "run_batch_fine_grained.py", line 283, in train optimizer.minimize(loss) File "/py37/lib/python3.7/site-packages/paddle/distributed/fleet/base/fleet_base.py", line 1501, in minimize loss, startup_program, parameter_list, no_grad_set=no_grad_set) File "/py37/lib/python3.7/site-packages/paddle/distributed/fleet/meta_optimizers/meta_optimizer_base.py", line 95, in minimize loss, startup_program, parameter_list, no_grad_set) File "/py37/lib/python3.7/site-packages/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py", line 516, in minimize_impl self._apply_sharding_pass(params_grads) File "/py37/lib/python3.7/site-packages/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py", line 295, in _apply_sharding_pass self._split_program(main_block) File "/py37/lib/python3.7/site-packages/paddle/distributed/fleet/meta_optimizers/sharding_optimizer.py", line 746, in _split_program assert (int(op.attr('op_role')) != int(OpRole.Optimize)) AssertionError