search

Justin1904 / TensorFusionNetworks

Pytorch Implementation of Tensor Fusion Networks for multimodal sentiment analysis.
169 stars 44 forks source link

Weights of audio sub network always converging towards 0 #5

Closed purvanshi closed 6 years ago

purvanshi commented 6 years ago

The last layer of audio subnetwork (audio_h in model.py file) always updates to a tensor of zeros, which means that audio input has no contribution towards the output. Is this what the implementation should be?

Justin1904 commented 6 years ago

Can you point me to exactly where the audio is being updated toward zeros values? Sorry I didn't quite figure out that part on my own.

purvanshi commented 6 years ago

check out model.py line 135. The output from the audio subnetwork. I tried printing the outputs of all sub networks and found that audio one always updates to zero.

Justin1904 commented 6 years ago

I guess this is probably because the default hyper parameters sets the dimensions of audio to 4, which limits the expressivity of audio modality, hence the model learns to just ignore it. Try different hyper parameter settings might help. Other than that, I don't see explicit regularization of the audio modality toward zero, so it could be that the audio modality just don't help as much after all.