search

jperezrua / mfas

Implementation of CVPR 2019 paper "Mfas: Multimodal fusion architecture search"
77 stars 20 forks source link

AVMNIST: my test acc is 65%! #17

Open mogvision opened 1 year ago

mogvision commented 1 year ago

Hi Thanks for sharing your nice work, I tried the AVmnist code for uni-modal image classification with different hyper-parameters, but I could not get results better than 65-6% while 75% acc is reported in the paper. Would you kindly guide me how to fix that? Thanks

bezirganyan commented 1 year ago

The dataset mentioned in the paper is not open-source, and the data generation code is not available as well. The dataset you used, is probably obtained from a third party, and hence you will not get the result mentioned in the paper.

I also got around 65% on that third party dataset, so your results seem correct.

I do, however, believe that it would be better if the authors at least open-sourced their data-generation code, to make their results reproducible.

mogvision commented 1 year ago

Many thanks for the update, I got it. Even in the case of audio, the default results with LeNet are worse and it is due to the dataset. I got 43% for audio-only which is too far from 66%. I also googled avmnist but there is no official repo for that. Thanks again and wish you lots of success.