Add DCAN - Githubissues

abheesht17 commented 2 years ago

NOTE: The DCAN default hyperparams don't seem to be the best choice of hyperparameters (since number of levels = 1, which means we don't have Dilated Conv layers of size 2, 4, 8, etc. We have only 1). I've opened an issue on the official DCAN repository.

NOTE: On further investigation, a 3-level DCAN (our implementation) has 8.9M params (the paper mentions that the model has 8.7M params. So, it's likely that they used a 3-level DCAN.

To-dos:

[x] Add layers (label_wise_attn, word_embedding_layer, temporal_block, temporal_conv_net)
[x] Add DCAN model class.
[x] Check if the model works (ran it for 11% of an epoch).
[x] Add doc-strings to functions and classes.
[x] DCAN uses label smoothing. I think we will have to write a custom loss class that inherits from PyTorch losses.
[x] Train model and verify results.
[x] Add log statements.
[x] Change dropout to scalar.

Do later:

[ ] Add a param, num_level. If num_level is specified, the user does not have to specify lists for the other model params. Model Structure:

DCAN(
(word_embedding_layer): WordEmbeddingLayer(
(embed): Embedding(49800, 100, padding_idx=0)
(dropout): Dropout(p=0.2, inplace=False)
)
(temporal_conv_net): TemporalConvNet(
(temporal_blocks): ModuleList(
  (0): TemporalBlock(
    (conv_temporal_sub_blocks): ModuleList(
      (0): ConvTemporalSubBlock(
        (conv_layer): Conv1d(100, 600, kernel_size=(2,), stride=(1,), padding=(1,))
        (chomp1d): Chomp1d()
        (activation): ReLU()
        (dropout): Dropout(p=0.2, inplace=False)
      )
      (1): ConvTemporalSubBlock(
        (conv_layer): Conv1d(600, 600, kernel_size=(2,), stride=(1,), padding=(1,))
        (chomp1d): Chomp1d()
        (activation): ReLU()
        (dropout): Dropout(p=0.2, inplace=False)
      )
    )
    (downsample): Conv1d(100, 600, kernel_size=(1,), stride=(1,))
    (output_activation): ReLU()
  )
  (1): TemporalBlock(
    (conv_temporal_sub_blocks): ModuleList(
      (0): ConvTemporalSubBlock(
        (conv_layer): Conv1d(600, 600, kernel_size=(2,), stride=(1,), padding=(1,))
        (chomp1d): Chomp1d()
        (activation): ReLU()
        (dropout): Dropout(p=0.2, inplace=False)
      )
      (1): ConvTemporalSubBlock(
        (conv_layer): Conv1d(600, 600, kernel_size=(2,), stride=(1,), padding=(1,))
        (chomp1d): Chomp1d()
        (activation): ReLU()
        (dropout): Dropout(p=0.2, inplace=False)
      )
    )
    (output_activation): ReLU()
  )
)
)
(linear_layer): Linear(in_features=600, out_features=300, bias=True)
(activation): ReLU()
(output_layer): OutputLayer(
(label_wise_attn): LabelWiseAttn(
  (U): Linear(in_features=300, out_features=50, bias=True)
)
(final): Linear(in_features=300, out_features=50, bias=True)
)
)

Results:

MIMIC-III (top-50, ours), with 2 level DCAN:

Val @ epoch 46:
INFO — Evaluate on val dataset
INFO —    prec_at_5: 0.678521
INFO —    prec_at_8: 0.549315
INFO —     macro_f1: 0.662994
INFO —     micro_f1: 0.733202
INFO —    macro_auc: 0.933964
INFO —    micro_auc: 0.952682
INFO — Checkpoint saved to ckpt-46.pth
INFO — Checkpoint saved to best-46.pth (prec_at_8: 0.549315)

Val @ epoch 56:
INFO — Evaluate on val dataset
INFO —    prec_at_5: 0.678266
INFO —    prec_at_8: 0.549156
INFO —     macro_f1: 0.663466
INFO —     micro_f1: 0.732965
INFO —    macro_auc: 0.934129
INFO —    micro_auc: 0.952837
INFO — Checkpoint saved to ckpt-56.pth

Test (@ epoch 46):
INFO — Loaded checkpoint from best-46.pth
INFO — Use GPU
INFO — Evaluating on test dataset
INFO —    prec_at_5: 0.679963
INFO —    prec_at_8: 0.553301
INFO —     macro_f1: 0.655410
INFO —     micro_f1: 0.727384
INFO —    macro_auc: 0.933417
INFO —    micro_auc: 0.952115
INFO — Saving result on results/dcan_mimic3_50/test_result.json

MIMIC-III (top-50, old), with 2 level DCAN (training not complete):

Val (@ epoch 26)
2022-04-01 12:41:14,544 — src.trainers.base_trainer — INFO —    prec_at_5: 0.642721
2022-04-01 12:41:14,551 — src.trainers.base_trainer — INFO —    prec_at_8: 0.517721
2022-04-01 12:41:14,563 — src.trainers.base_trainer — INFO —     macro_f1: 0.564367
2022-04-01 12:41:14,574 — src.trainers.base_trainer — INFO —     micro_f1: 0.668651
2022-04-01 12:41:14,643 — src.trainers.base_trainer — INFO —    macro_auc: 0.906983
2022-04-01 12:41:14,663 — src.trainers.base_trainer — INFO —    micro_auc: 0.932760

Test (@ epoch 26)
2022-04-01 12:47:00,561 — src.trainers.base_trainer — INFO —    prec_at_5: 0.642684
2022-04-01 12:47:00,562 — src.trainers.base_trainer — INFO —    prec_at_8: 0.523641
2022-04-01 12:47:00,562 — src.trainers.base_trainer — INFO —     macro_f1: 0.560526
2022-04-01 12:47:00,562 — src.trainers.base_trainer — INFO —     micro_f1: 0.665048
2022-04-01 12:47:00,563 — src.trainers.base_trainer — INFO —    macro_auc: 0.905683
2022-04-01 12:47:00,563 — src.trainers.base_trainer — INFO —    micro_auc: 0.932133

abheesht17 commented 2 years ago

Thank you, @dalgu90, for running these configurations!

3-level DCAN results (comfortably beats the results in the paper, except for Macro F1):

Paper Results:

prec_at_5: 0.642
macro_f1: 0.615
micro_f1: 0.671
macro_auc: 0.902
micro_auc: 0.931

MIMIC-III top-50 [old]:

2022-04-04 10:47:09,139 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50_old/test.json (1729 examples)
2022-04-04 10:47:09,140 — src.models.dcan — INFO — Initialising DCAN
2022-04-04 10:47:09,324 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-04 10:47:12,576 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-48.pth
2022-04-04 10:47:12,584 — src.trainers.base_trainer — INFO — Use GPU
2022-04-04 10:47:12,584 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —    prec_at_5: 0.649393
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —    prec_at_8: 0.533835
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —     macro_f1: 0.611135
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —     micro_f1: 0.688379
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —    macro_auc: 0.913397
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO —    micro_auc: 0.937868
2022-04-04 10:47:17,281 — src.trainers.base_trainer — INFO — Saving result on results/dcan_mimic3_50_old/test_result.json

MIMIC-III top-50:

2022-04-05 00:03:25,129 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_50/test.json (3234 examples)
2022-04-05 00:03:25,129 — src.models.dcan — INFO — Initialising DCAN
2022-04-05 00:03:25,340 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-05 00:03:28,528 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-35.pth
2022-04-05 00:03:28,539 — src.trainers.base_trainer — INFO — Use GPU
2022-04-05 00:03:28,539 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —    prec_at_5: 0.681014
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —    prec_at_8: 0.553224
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —     macro_f1: 0.658478
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —     micro_f1: 0.727083
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —    macro_auc: 0.933557
2022-04-05 00:03:35,965 — src.trainers.base_trainer — INFO —    micro_auc: 0.953008
2022-04-05 00:03:35,966 — src.trainers.base_trainer — INFO — Saving result on results/dcan_mimic3_50/test_result.json

MIMIC-III full [old]:

2022-04-06 05:20:36,302 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full_old/test.json (3372 examples)
2022-04-06 05:20:36,302 — src.models.dcan — INFO — Initialising DCAN
2022-04-06 05:20:36,576 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-06 05:20:39,890 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-48.pth
2022-04-06 05:20:39,907 — src.trainers.base_trainer — INFO — Use GPU
2022-04-06 05:20:39,907 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-06 05:25:10,784 — src.trainers.base_trainer — INFO —    prec_at_8: 0.722160
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO —   prec_at_15: 0.570087
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO —     macro_f1: 0.063707
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO —     micro_f1: 0.524617
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO —    macro_auc: 0.844949
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO —    micro_auc: 0.978251
2022-04-06 05:25:10,785 — src.trainers.base_trainer — INFO — Saving result on results/dcan_mimic3_full_old/test_result.json

MIMIC-III full:

2022-04-06 02:48:46,230 — src.datasets.base_dataset — INFO — Loaded dataset from datasets/mimic3_full/test.json (3372 examples)
2022-04-06 02:48:46,231 — src.models.dcan — INFO — Initialising DCAN
2022-04-06 02:48:46,475 — src.trainers.base_trainer — INFO — Initializing BaseTrainer
2022-04-06 02:48:49,775 — src.trainers.base_trainer — INFO — Loaded checkpoint from best-31.pth
2022-04-06 02:48:49,807 — src.trainers.base_trainer — INFO — Use GPU
2022-04-06 02:48:49,807 — src.trainers.base_trainer — INFO — Evaluating on test dataset
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —    prec_at_8: 0.718491
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —   prec_at_15: 0.568149
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —     macro_f1: 0.057142
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —     micro_f1: 0.522896
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —    macro_auc: 0.854977
2022-04-06 02:55:54,542 — src.trainers.base_trainer — INFO —    micro_auc: 0.980384
2022-04-06 02:55:54,543 — src.trainers.base_trainer — INFO — Saving result on results/dcan_mimic3_full/test_result.json

abheesht17 commented 2 years ago

@dalgu90, the weirdest thing is happening. I am pushing to my branch, but the commits are not showing up on this PR 🤣 . I think I'll merge this one, and open another PR for the rest of the changes.

dalgu90 / icd-coding-benchmark

Add DCAN #31