Unable to replicate paper results

[purvanshi]

Hey, I am unable to replicate the results (get a binary test accuracy of 77.1) I made the following changes of hyperparameters in your code -

1) Changed the output of the fusion layer to 128 instead of 32 2) dropout of 0.15 as stated in the paper 3) added l2 norm regulariser by adding a weight decaying factor of 0.01 4) Changed the number of output nodes of the text modality to 64 instead of 32

I run the model with a batch_size of 128, 1000 epochs and a patience of 100 Am I missing something here?

[Justin1904]

Hi, I think the main reason is actually the TFN paper used a data split that was different than all the later papers (as well as the one the CMU-Multimodal SDK is offering). You can check out the latest TFN performance on the current data split here: https://arxiv.org/pdf/1802.00923.pdf It is a paper by the same authors.

[ghost]

Hello @purvanshi. For the most updated results of the Tensor Fusion, please check the following: https://arxiv.org/pdf/1808.03920.pdf

If you are not able to replicate the results still, please be in touch directly with us.

[purvanshi]

Hello @A2Zadeh, I am still not able to retrieve the 74% accuracy. Do you have a newer version of the code with the hyperparameters set? Thanks for the help.

[ghost]

Ok. Can you check out this code: https://github.com/A2Zadeh/TensorFusionNetwork

Let me know how it works.

[purvanshi]

I had some issues while running that code, have raised an issue there.

Thanks, Purvanshi

[purvanshi]

Hey @Justin1904 I am still not able to get the accuracy of the new paper - https://arxiv.org/pdf/1802.00923.pdf . I would be grateful if you could help me out. I am able to get a validation accuracy of 72.5 Thanks, Purvanshi

[Justin1904]

@purvanshi Which one of the experiments are you referring to?

[purvanshi]

I am trying to run TFN on MOSI dataset.

[Justin1904]

I'll try running some grid search and get back to you. I think I was able to achieve something around 74 F1 with enough grid search.

[Justin1904]

Wait, I just found a bug in the code: in train.py, when evaluating the model after training, I forgot to use the loaded best model but instead used the most recent model (which probably overfits train/valid set).

I'll update the code in a bit, but feel free to just change model to best_model at

https://github.com/Justin1904/TensorFusionNetworks/blob/0662ba1518343326ac717e95ad82f0e259a916d2/train.py#L173

[purvanshi]

sure, thanks a lot. I will put it on training and get back to you.

[purvanshi]

These are the results -

[Justin1904]

I see. Probably some grid search is needed here. I'll run some grid search on the hyperparameters and update them as well as the best model when I get some comparable results.

[purvanshi]

thanks a lot!! Looking forward to it.

[purvanshi]

Hey @Justin1904 did you get the desired accuracy? Thanks!!

[Justin1904]

Still no luck. I've been a bit busy and wasn't able to take care of this fully and only let the grid search ran, but so far I myself haven't got a good result either. Can you help me check if model-wise this PyTorch implementation is doing something very different thant the Keras one here

https://github.com/A2Zadeh/TensorFusionNetwork

?

[purvanshi]

Hey, Nothing that fundamentally different but some hyperparameters were changed which I have mentioned here - https://github.com/A2Zadeh/TensorFusionNetwork/issues/5 I tried certain things but still couldn't get the results.

[ghost]

@purvanshi I am sending this to Paul Pu Liang since he knows where the hyperparameters for TFN are. Sorry for all the back and forth @purvanshi. We are a bit congested these days with work. Hopefully we will resolve this issue soon.

[kushagrajain11]

hey @purvanshi, are you running this code with the new mmsdk api? Could you help me run this code? Thanks!

[purvanshi]

@A2Zadeh Thanks a lt. I will connect to him. @kushagrajain11 what is the issue you are facing?

[kushagrajain11]

The mmdata file used in utils is no longer available. I want to know how you create the dataset for this code from mmsdk.

[purvanshi]

@kushagrajain11 I am using the older mmdata file version to run it. Have not checked out the new version.

[kushagrajain11]

Could you share the file and the data somehow? It would be really helpful.

[ghost]

@kushagrajain11 we are currently using a new version of the SDK. I highly suggest you move to the new one. It requires small modifications to make it work with this code but we will use that in long term. We could bring back the previous implementation of Multimodal SDK for you just in case.

[kushagrajain11]

@A2Zadeh Ill try working with the new Multimodal SDK. According to my understanding if I convert the h5py file format into numpy arrays it would work with this code. Is that right?

[purvanshi]

Hey @A2Zadeh I havnt heard from Paul Pu liang yet is there anyone else who can help me with the hyperparameters?

[hjs542761058]

@Justin1904 Do you have a file named mmdata.py ?

[Justin1904]

@hjs542761058 No, but I remember having some folders or modules in the current SDK that has the name mmdata. Could you point me to where you came across that? Any code snippet that uses it?

[hjs542761058]

@Justin1904 I can't find the mmdata in current SDK(Version 1.0.3). I made an error while constructing a dataloader.

instantiate a multimodal dataloader

self.dataloader = mmdata.dict[dataset]()

[Justin1904]

@hjs542761058 I see. I think this is the API for the old SDK, which we no longer maintain. You probably need to write your own data processing code using the new SDK in this case. We're planning to update the code in this repo, but we probably won't be able to finish it very soon.