Add the MultiResCNN Model

dalgu90 / icd-coding-benchmark

Automatic ICD coding benchmark based on the MIMIC dataset

MIT License

35 stars 5 forks source link

Edit: I added the results of all 4 MIMIC-III datasets. The top-50 (old) dataset results seems lower than what is in the MultiResCNN paper. (Maybe we need to check whether we early stop on P@5 or P@8?)

MIMIC-III top-50 (old) (P@5: 0.638, Micro-F1: 0.673, Macro-F1: 0.608 in the paper)

2022-04-24 15:07:56,531 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50_old/test.json (1729 examples)
loading pretrained embeddings from processed_full.embed
2022-04-24 15:07:58,061 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-24 15:08:07,340 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-45.pth
2022-04-24 15:08:07,767 — src.trainers.base_trainer — INFO — Use GPU
2022-04-24 15:08:07,767 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-24 15:08:15,600 — src.trainers.base_trainer — INFO —    prec_at_5: 0.625795
2022-04-24 15:08:15,600 — src.trainers.base_trainer — INFO —    prec_at_8: 0.510628
2022-04-24 15:08:15,600 — src.trainers.base_trainer — INFO —     macro_f1: 0.591825
2022-04-24 15:08:15,600 — src.trainers.base_trainer — INFO —     micro_f1: 0.659234
2022-04-24 15:08:15,600 — src.trainers.base_trainer — INFO —    macro_auc: 0.896389
2022-04-24 15:08:15,601 — src.trainers.base_trainer — INFO —    micro_auc: 0.923049
2022-04-24 15:08:15,601 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_50_old/test_result.json

MIMIC-III top-50 (new)

2022-04-27 08:24:50,470 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50/test.json (3234 examples)
loading pretrained embeddings from processed_full.embed
2022-04-27 08:24:51,766 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-27 08:24:55,835 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-58.pth
2022-04-27 08:24:55,845 — src.trainers.base_trainer — INFO — Use GPU
2022-04-27 08:24:55,845 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —    prec_at_5: 0.664997
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —    prec_at_8: 0.543329
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —     macro_f1: 0.646047
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —     micro_f1: 0.715030
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —    macro_auc: 0.920296
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO —    micro_auc: 0.946453
2022-04-27 08:25:00,732 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_50/test_result.json

MIMIC-III full (old) (P@8: 0.734, Micro-F1: 0.552, Macro-F1: 0.552 in the paper)

2022-04-26 09:11:08,265 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full_old/test.json (3372 examples)
loading pretrained embeddings from processed_full.embed
2022-04-26 09:11:09,739 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-26 09:11:16,072 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-99.pth
2022-04-26 09:11:16,092 — src.trainers.base_trainer — INFO — Use GPU
2022-04-26 09:11:16,092 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —    prec_at_5: 0.813049
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —    prec_at_8: 0.736321
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —     macro_f1: 0.074447
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —     micro_f1: 0.551759
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —    macro_auc: 0.899630
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO —    micro_auc: 0.985819
2022-04-26 09:17:07,605 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_full_old/test_result.json

MIMIC-III full (new)

2022-04-26 02:40:13,937 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full/test.json (3372 examples)
loading pretrained embeddings from processed_full.embed
2022-04-26 02:40:15,429 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-26 02:40:21,823 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-94.pth
2022-04-26 02:40:21,977 — src.trainers.base_trainer — INFO — Use GPU
2022-04-26 02:40:21,977 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —    prec_at_5: 0.811625
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —    prec_at_8: 0.735765
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —     macro_f1: 0.072757
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —     micro_f1: 0.552928
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —    macro_auc: 0.895491
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO —    micro_auc: 0.986007
2022-04-26 02:55:11,142 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_full/test_result.json

2022-05-27 11:34:12,805 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50_old/test.json (1729 examples) loading pretrained embeddings from processed_full.embed 2022-05-27 11:34:14,014 — src.trainers.base_trainer — INFO — Initializing BaseTrainer 2022-05-27 11:34:18,348 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-14.pth 2022-05-27 11:34:18,355 — src.trainers.base_trainer — INFO — Use GPU 2022-05-27 11:34:18,356 — src.trainers.base_trainer — INFO — Evaluating on test dataset 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — prec_at_5: 0.640023 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — prec_at_8: 0.519158 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — macro_f1: 0.589514 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — micro_f1: 0.670533 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — macro_auc: 0.899234 2022-05-27 11:34:21,189 — src.trainers.base_trainer — INFO — micro_auc: 0.927863 2022-05-27 11:34:21,190 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_50_old/test_result.json

2022-05-28 03:36:48,582 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50/test.json (3234 examples) 2022-05-28 03:36:49,643 — src.trainers.base_trainer — INFO — Initializing BaseTrainer 2022-05-28 03:36:55,193 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-7.pth 2022-05-28 03:36:55,288 — src.trainers.base_trainer — INFO — Use GPU 2022-05-28 03:36:55,288 — src.trainers.base_trainer — INFO — Evaluating on test dataset 2022-05-28 03:37:03,231 — src.trainers.base_trainer — INFO — prec_at_5: 0.672604 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — prec_at_8: 0.549204 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — macro_f1: 0.644801 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — micro_f1: 0.718600 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — macro_auc: 0.926459 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — micro_auc: 0.949508 2022-05-28 03:37:03,232 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_50/test_result.json

2022-05-28 10:03:43,204 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full_old/test.json (3372 examples) 2022-05-28 10:03:44,408 — src.trainers.base_trainer — INFO — Initializing BaseTrainer 2022-05-28 10:03:47,759 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-25.pth 2022-05-28 10:03:47,774 — src.trainers.base_trainer — INFO — Use GPU 2022-05-28 10:03:47,774 — src.trainers.base_trainer — INFO — Evaluating on test dataset 2022-05-28 10:04:27,896 — src.trainers.base_trainer — INFO — prec_at_8: 0.736099 2022-05-28 10:04:27,896 — src.trainers.base_trainer — INFO — prec_at_15: 0.583472 2022-05-28 10:04:27,896 — src.trainers.base_trainer — INFO — macro_f1: 0.077205 2022-05-28 10:04:27,896 — src.trainers.base_trainer — INFO — micro_f1: 0.550036 2022-05-28 10:04:27,896 — src.trainers.base_trainer — INFO — macro_auc: 0.907786 2022-05-28 10:04:27,897 — src.trainers.base_trainer — INFO — micro_auc: 0.986263 2022-05-28 10:04:27,897 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_full_old/test_result.json

2022-05-28 09:41:42,509 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full/test.json (3372 examples) 2022-05-28 09:41:43,718 — src.trainers.base_trainer — INFO — Initializing BaseTrainer 2022-05-28 09:41:47,032 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-34.pth 2022-05-28 09:41:47,047 — src.trainers.base_trainer — INFO — Use GPU 2022-05-28 09:41:47,047 — src.trainers.base_trainer — INFO — Evaluating on test dataset 2022-05-28 09:42:26,531 — src.trainers.base_trainer — INFO — prec_at_8: 0.739324 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — prec_at_15: 0.587169 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — macro_f1: 0.084015 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — micro_f1: 0.558163 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — macro_auc: 0.914338 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — micro_auc: 0.986860 2022-05-28 09:42:26,532 — src.trainers.base_trainer — INFO — Saving result on results/MultiResCNN_mimic3_full/test_result.json

dalgu90 / icd-coding-benchmark

Add the MultiResCNN Model #24