Multi-GPU when training Fastspeech2

[ZhaoZeqing]

Hi guys,

I use python3.7.9, tensorflow 2.3.1, cuda-10.1, 1080 Ti, It's ok when training on one GPU. But when training Fastspeech2 on multi-gpu, it got this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: 3 root error(s) found. (0) Invalid argument: Incompatible shapes: [16,2,99,99] vs. [16,1,1,81] [[node replica_1/tf_fast_speech2end/encoder/layer._0/attention/self/add (defined at ./tensorflow_tts/models/fastspeech.py:246) ]] [[tf_fast_speech2_end/length_regulator/while/loop_body_control/_89/_279]] (1) Invalid argument: Incompatible shapes: [16,2,99,99] vs. [16,1,1,81] [[node replica_1/tf_fast_speech2end/encoder/layer._0/attention/self/add (defined at ./tensorflow_tts/models/fastspeech.py:246) ]] [[replica_1/cond_2/batch_3/_529/_874]] (2) Invalid argument: Incompatible shapes: [16,2,99,99] vs. [16,1,1,81] [[node replica_1/tf_fast_speech2end/encoder/layer._0/attention/self/add (defined at ./tensorflow_tts/models/fastspeech.py:246) ]] 0 successful operations. 0 derived errors ignored. [Op:inferenceone_step_forward_68611]

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.