search

TensorSpeech / TensorFlowTTS

:stuck_out_tongue_closed_eyes: TensorFlowTTS: Real-Time State-of-the-art Speech Synthesis for Tensorflow 2 (supported including English, French, Korean, Chinese, German and Easy to adapt for other languages)
https://tensorspeech.github.io/TensorFlowTTS/
Apache License 2.0
3.8k stars 810 forks source link

How can I get intermediate weights/gradients/layer_outputs while training? #656

Closed biofoolgreen closed 2 years ago

biofoolgreen commented 3 years ago

Hi, I'm debugging a numerial issue on FastSpeech2. I'd like to printing/saving weights/gradients/outputs of each layer. I've tried add logger to _calculate_gradient_per_batch function in base_trainer.py but only got tensors not numpy values.

How could I get intermediate weights/gradients of each layer while training? It's very appreciated if you have any convienient way to debug numerical issue. Thanks in advance.

dathudeptrai commented 3 years ago

can you tell me what exactly a bug is?

biofoolgreen commented 3 years ago

@dathudeptrai The issue is my loss curve looks good, but when I do inference with trained weights, it's really bad that give me all noise. The experiments were not running on GPU but other hardware(Graphcore IPU). So I want to check if all weights/gradients/layer_outputs on IPU are identical to GPU. I believe this is more like a hardware related bug than this repo. But I don't know how to get what I need here.

biofoolgreen commented 3 years ago

The mel-spectrum I got like this: wrong_mel

Any suggestion?

dathudeptrai commented 3 years ago

@biofoolgreen are the eval mels good ?

biofoolgreen commented 3 years ago

@dathudeptrai See above mels. Even I use trained sample is bad too.

dathudeptrai commented 3 years ago

@biofoolgreen can you share your tensorboard ?

biofoolgreen commented 3 years ago

@dathudeptrai Sure! Here is the tensorboard zip file. tensorboard.zip

EmreOzkose commented 2 years ago

@biofoolgreen did you reach output of mid-layers (feature maps)? I wanna reach too, to just observe.

EmreOzkose commented 2 years ago

we can do something like that for multiband-melgan:

intermadiate_layer_index = 20 (the layer we want)
feature_extactor = mb_melgan.get_layer(index=0)
inter_output_model = tf.keras.Model(feature_extactor.input, feature_extactor.get_layer(index = intermadiate_layer_index).output)

feat = mel_spectrogram    # (1, #frame, 80)
intermediate_output = inter_output_model(feat)    # (1, #frame, #feat_dimension)
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.