Implement Fusion model

[dalgu90]

Hello guys.

I ported the Fusion model. You can see the paper's hyper-parameters in the config files (configs/fusion). Paper: https://aclanthology.org/2021.findings-acl.184/ GitHub repo: https://github.com/machinelearning4health/Fusion

NOTE: In the paper, the performance on the MIMIC-III top-50 (equivalent to our top-50 old dataset) was made by training on the full dataset (with 5x examples). This dataset is equivalent to our new top-50 dataset, so we compared it below. The authors now share the results on the original top-50 dataset, which we compared in the MIMIC-III top-50 (old).

• MIMIC-III top-50 (old)

Code	Macro AUC	Micro AUC	Macro F1	Micro F1	P@5	Note
Author	0.894	0.924	0.598	0.659	0.635	Updated in the repo
Ours	0.904610	0.929229	0.611743	0.674127	0.640023

• MIMIC-III top-50 (ours)

Code	Macro AUC	Micro AUC	Macro F1	Micro F1	P@5	Note
Author	0.931	0.950	0.683	0.725	0.679	Top-50 in the paper (or enhanced 50 settings in the repo)
Ours	0.932295	0.952335	0.660386	0.726407	0.678726

• MIMIC-III full (old)

Code	Macro AUC	Micro AUC	Macro F1	Micro F1	P@8	Note
Author	0.915	0.987	0.083	0.554	0.736	MIMIC-III full In the paper
Ours	0.907964	0.986258	0.079416	0.559838	0.747628

• MIMIC-III full (ours)

Code	Macro AUC	Micro AUC	Macro F1	Micro F1	P@8	Note
Ours	0.912643	0.986653	0.078532	0.556019	0.743105

Here's the raw output of our implementation:

• MIMIC-III top-50 (old)

  2022-04-25 23:09:49,876 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50_old/test.json (1729 examples)
  2022-04-25 23:09:49,877 — src.models.fusion — INFO — Initialising Fusion
  2022-04-25 23:09:52,828 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
  2022-04-25 23:10:01,660 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-22.pth
  2022-04-25 23:10:01,942 — src.trainers.base_trainer — INFO — Use GPU
  2022-04-25 23:10:01,943 — src.trainers.base_trainer — INFO — Evaluating on test dataset
  2022-04-25 23:10:20,706 — src.trainers.base_trainer — INFO —    prec_at_5: 0.640023
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO —    prec_at_8: 0.519881
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO —     macro_f1: 0.611743
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO —     micro_f1: 0.674127
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO —    macro_auc: 0.904610
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO —    micro_auc: 0.929229
  2022-04-25 23:10:20,707 — src.trainers.base_trainer — INFO — Saving result on results/Fusion_mimic3_50_old/test_result.json

• MIMIC-III top-50 (ours)

  2022-04-26 00:39:03,049 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50/test.json (3234 examples)
  2022-04-26 00:39:03,049 — src.models.fusion — INFO — Initialising Fusion
  2022-04-26 00:39:06,165 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
  2022-04-26 00:39:11,777 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-10.pth
  2022-04-26 00:39:11,790 — src.trainers.base_trainer — INFO — Use GPU
  2022-04-26 00:39:11,791 — src.trainers.base_trainer — INFO — Evaluating on test dataset
  2022-04-26 00:39:28,331 — src.trainers.base_trainer — INFO —    prec_at_5: 0.678726
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO —    prec_at_8: 0.553030
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO —     macro_f1: 0.660386
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO —     micro_f1: 0.726407
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO —    macro_auc: 0.932295
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO —    micro_auc: 0.952335
  2022-04-26 00:39:28,332 — src.trainers.base_trainer — INFO — Saving result on results/Fusion_mimic3_50/test_result.json

• MIMIC-III full (old)

  2022-04-28 13:13:21,992 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full_old/test.json (3372 examples)
  2022-04-28 13:13:21,992 — src.models.fusion — INFO — Initialising Fusion
  2022-04-28 13:13:25,253 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
  2022-04-28 13:13:31,757 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-32.pth
  2022-04-28 13:13:31,928 — src.trainers.base_trainer — INFO — Use GPU
  2022-04-28 13:13:31,928 — src.trainers.base_trainer — INFO — Evaluating on test dataset
  2022-04-28 13:19:29,970 — src.trainers.base_trainer — INFO —    prec_at_8: 0.747628
  2022-04-28 13:19:29,970 — src.trainers.base_trainer — INFO —   prec_at_15: 0.591281
  2022-04-28 13:19:29,970 — src.trainers.base_trainer — INFO —     macro_f1: 0.079416
  2022-04-28 13:19:29,971 — src.trainers.base_trainer — INFO —     micro_f1: 0.559838
  2022-04-28 13:19:29,971 — src.trainers.base_trainer — INFO —    macro_auc: 0.907964
  2022-04-28 13:19:29,971 — src.trainers.base_trainer — INFO —    micro_auc: 0.986258
  2022-04-28 13:19:29,971 — src.trainers.base_trainer — INFO — Saving result on results/Fusion_mimic3_full_old/test_result.json

• MIMIC-III full (ours)

  2022-04-28 13:13:16,204 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full/test.json (3372 examples)
  2022-04-28 13:13:16,205 — src.models.fusion — INFO — Initialising Fusion
  2022-04-28 13:13:19,548 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
  2022-04-28 13:13:25,908 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-32.pth
  2022-04-28 13:13:26,078 — src.trainers.base_trainer — INFO — Use GPU
  2022-04-28 13:13:26,078 — src.trainers.base_trainer — INFO — Evaluating on test dataset
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —    prec_at_8: 0.743105
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —   prec_at_15: 0.588988
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —     macro_f1: 0.078532
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —     micro_f1: 0.556019
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —    macro_auc: 0.912643
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO —    micro_auc: 0.986653
  2022-04-28 13:19:36,959 — src.trainers.base_trainer — INFO — Saving result on results/Fusion_mimic3_full/test_result.json

[abheesht17]

@dalgu90, forgot to mention one thing - please add the results for Fusion to the README in this PR itself

[dalgu90]

Thanks @abheesht17 for the detailed review. Let me fix them in the later commit (after our next meeting).

[abheesht17]

Thanks @abheesht17 for the detailed review. Let me fix them in the later commit (after our next meeting).

Sure! :)

[dalgu90]

Hey team. I reflected the comments except the ones with fixing the class/layer names.