search

SuperSupermoon / MedViLL

MedViLL official code. (Published IEEE JBHI 2021)
MIT License
83 stars 11 forks source link

Cannot achieve the various metrics described in the paper after running the code. #6

Closed PengPeixi closed 2 years ago

PengPeixi commented 2 years ago

Hello, your code cannot achieve the various metrics described in the paper after running. Could you please provide the hyperparameters you set before running, and the training log after running.

SuperSupermoon commented 2 years ago

Hello, PengPeixi. Thank you for your interest. We will provide hyperparameters and training logs soon. In the meantime, for which tasks in the paper were you unable to achieve metrics? We would appreciate it if you could leave a specific issue.

PengPeixi commented 2 years ago

For example, for the downstream task of medical report generation, after running the file generation_decode.py on the openi dataset, the metrics obtained by the MedViLL model are: {'bleu 1': 0.15519337010503506, 'bleu 2': 0.09391290091144913, 'bleu 3': 0.05791537394797654, 'bleu 4': 0.03430614826591653, 'best_bleu1': 0.15519337010503506, 'best_bleu2': 0.09391290091144913, 'best_bleu3': 0.05791537394797654, 'best_bleu4': 0.03430614826591653} It doesn't reach the level of the paper.

SuperSupermoon commented 2 years ago

Hello, PengPeixi. Thanks for you patience. Due to code refactoring work, it has been delayed to answer. Note that the models released in this GitHub repository are pre-trained models. Therefore, to reach the performance of our paper, the pre-trained model needs to be a fine-tuning process. We will release our refactored code v.2 soon, and we will additionally reveal the performance achieved when using multiple CNN backbones soon. Please stay tuned!