[exp] t5 + diaASQ [en]

[Nana2929]

T5-Generation Fine-tuning Task

Data Preparation

• can only use specially-preprocessed: speaker-spec dataset, because the max input length for t5 is 512.
• preprocessing
  1. lib/create_speaker_data.py --cfg=configs/diaasq-t5-en.yaml for creating speaker-specific data. Note that do not use speaker-spec configs. The resulted data will be saved to /home/nanaeilish/projects/research/sentiment-llm/data/diaasq/speaker_dataset/jsons_en (take en as lang_src for example; modify config if needed).
  2. lib/create_kshot_dataset_split.py --cfg=configs/diaasq-t5-speaker-spec-en.yaml --is_speaker_dataset for creating k-shot in-context examples for speaker-specific data. The script will reuse the data created above and then save resulted data to data/diaasq/speaker_dataset/proc/jsons_en.
  3. Don't forget to set speaker in step 2. so that the in-context examples all contain the same speaker.
• prompts need extra designed
  • To be precise, "in-context examples design"
    1. Filter trainset, leaving the speaker data example with 3 sentiment tuples (complicated, demonstrative enough)
    2. Sort the data example with the number of triplets; choose the k shots with the fewest triplets. This is because T5 has very strict input length limit (512), which is too short for a data example to be filled in... (hence k = 1 in the below experiment),

Eval metrics (micro)

• target, aspect, opinion F1
• $pair{t-a}, pair{t-o}, pair_{a-o} F1$
• quadruple F1

  Loss

  order does not matter; but (needs inspection) loss seems to matter for seq2seq training loss: loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1))

[Nana2929]

Configs

# reference: https://tsmatz.wordpress.com/2022/11/25/huggingface-japanese-summarization/
# Note : Do not use FP16 precision in mT5 fine-tuning.
seed: 42
data:
  data_root: 'data/diaasq/speaker_dataset'
  train_split_name: 'train'
  test_split_name: 'valid'
  lang_src : 'en'

proc_data:
  data_root: 'data/diaasq/speaker_dataset/proc'
  train_ic_name: 't5_in_context' # use t5/create_kshot_dataset_split.py
  t5_train_split_name: 't5_train'
  test_ic_name: 't5_in_context'  # use the same in-context examples as in training
  t5_test_split_name: 't5_valid'

dataset:
  k: 1 # number of in-context examples
  prompt_path: 'prompt/experiment/diaasq-speaker-spec-en-t5'
  in_context_strategy: None

model:
  model_name: 'allenai/tk-instruct-base-def-pos'
  max_length: 512
  output_dir: 'output/diaasq/t5-en'

tokenizer:
  truncation_side: 'left'

trainer:
  gpu_id: "1"  # if training on server 9102, only 1 gpu at a time can be used otherwise it crashes
  push_to_hub: false
  per_device_train_batch_size: 4
  per_device_eval_batch_size: 2
  epochs: 20
  weight_decay: 0.01
  learning_rate: 0.00004
  # https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/trainer_utils.py#L362
  lr_scheduler_type: 'cosine_with_restarts'
  warmup_steps: 100
  optim : 'adamw_hf'
  generation_max_length: 128
  save_strategy: 'epoch'
  load_best_model_at_end: true
  metric_for_best_model: 'eval_quad_micro_f1'
  logging_steps: 500
  save_total_limit: 2

logger:
  logging_dir: 'logs/diaasq/t5-en' # logs/diaasq/t5-en/exp_20230524-143733
  report_to: 'tensorboard'
  run_name: 'exp'

Results

有稍微調過參數（用 ReducedOnPlateau 的 Scheduler）

trainset length: 2236
testset length: 290
[tensorboard] logging to: logs/diaasq/t5-en/exp_20230524-143733
{'loss': 2.7081, 'learning_rate': 3.855595667870036e-05, 'epoch': 0.89}
{'eval_loss': 0.3480522930622101, 'eval_target_f1': 0.46002805044075473, 'eval_aspect_f1': 0.40658049348693526, 'eval_opinion_f1': 0.346368715033774, 'eval_pair_ta_f1': 0.2229102166684056, 'eval_pair_to_f1': 0.1981424148108132, 'eval_pair_ao_f1': 0.19875776392520864, 'eval_quad_micro_f1': 0.09233791743557149, 'eval_runtime': 20.0959, 'eval_samples_per_second': 14.431, 'eval_steps_per_second': 7.215, 'epoch': 1.0}
{'loss': 0.2968, 'learning_rate': 3.675090252707581e-05, 'epoch': 1.79}
{'eval_loss': 0.3033939301967621, 'eval_target_f1': 0.5027624308891013, 'eval_aspect_f1': 0.4617142856641855, 'eval_opinion_f1': 0.3844492440103924, 'eval_pair_ta_f1': 0.27794561928535355, 'eval_pair_to_f1': 0.23176823171821667, 'eval_pair_ao_f1': 0.24799999994995361, 'eval_quad_micro_f1': 0.13523809518818286, 'eval_runtime': 9.1547, 'eval_samples_per_second': 31.678, 'eval_steps_per_second': 15.839, 'epoch': 2.0}
{'loss': 0.3704, 'learning_rate': 3.494584837545127e-05, 'epoch': 2.68}
{'eval_loss': 0.28762921690940857, 'eval_target_f1': 0.5572413792602071, 'eval_aspect_f1': 0.4582843712777574, 'eval_opinion_f1': 0.4099099098598998, 'eval_pair_ta_f1': 0.2848232847734609, 'eval_pair_to_f1': 0.26375908613915194, 'eval_pair_ao_f1': 0.25078369900965075, 'eval_quad_micro_f1': 0.15476190471229057, 'eval_runtime': 9.0573, 'eval_samples_per_second': 32.018, 'eval_steps_per_second': 16.009, 'epoch': 3.0}
{'loss': 0.2206, 'learning_rate': 3.314079422382672e-05, 'epoch': 3.58}
{'eval_loss': 0.2941332459449768, 'eval_target_f1': 0.5564971750910873, 'eval_aspect_f1': 0.47990543730216034, 'eval_opinion_f1': 0.43230944249838305, 'eval_pair_ta_f1': 0.3039832284617675, 'eval_pair_to_f1': 0.27643979052613465, 'eval_pair_ao_f1': 0.2776025236094272, 'eval_quad_micro_f1': 0.16350947153567927, 'eval_runtime': 8.9849, 'eval_samples_per_second': 32.276, 'eval_steps_per_second': 16.138, 'epoch': 4.0}
{'loss': 0.2011, 'learning_rate': 3.133574007220217e-05, 'epoch': 4.47}
{'eval_loss': 0.278605192899704, 'eval_target_f1': 0.551626591180398, 'eval_aspect_f1': 0.4941995359127635, 'eval_opinion_f1': 0.45879732734414913, 'eval_pair_ta_f1': 0.31115660179244714, 'eval_pair_to_f1': 0.2827868851959749, 'eval_pair_ao_f1': 0.2986611739973849, 'eval_quad_micro_f1': 0.17170731702340492, 'eval_runtime': 8.9753, 'eval_samples_per_second': 32.311, 'eval_steps_per_second': 16.156, 'epoch': 5.0}
{'loss': 0.1837, 'learning_rate': 2.953068592057762e-05, 'epoch': 5.37}
{'eval_loss': 0.2796546220779419, 'eval_target_f1': 0.5495750707713776, 'eval_aspect_f1': 0.5234741783536466, 'eval_opinion_f1': 0.4527027026526829, 'eval_pair_ta_f1': 0.33367875642682787, 'eval_pair_to_f1': 0.2880829015045442, 'eval_pair_ao_f1': 0.3041666666167283, 'eval_quad_micro_f1': 0.18163869689011578, 'eval_runtime': 8.9184, 'eval_samples_per_second': 32.517, 'eval_steps_per_second': 16.259, 'epoch': 6.0}
{'loss': 0.1756, 'learning_rate': 2.772563176895307e-05, 'epoch': 6.26}
{'eval_loss': 0.2843130826950073, 'eval_target_f1': 0.5605633802315326, 'eval_aspect_f1': 0.5174825174324013, 'eval_opinion_f1': 0.4864864864364591, 'eval_pair_ta_f1': 0.3302373580512523, 'eval_pair_to_f1': 0.3061013443141352, 'eval_pair_ao_f1': 0.3204994796586897, 'eval_quad_micro_f1': 0.1793103447779015, 'eval_runtime': 8.825, 'eval_samples_per_second': 32.861, 'eval_steps_per_second': 16.431, 'epoch': 7.0}
{'loss': 0.1555, 'learning_rate': 2.5920577617328523e-05, 'epoch': 7.16}
{'eval_loss': 0.28299134969711304, 'eval_target_f1': 0.5643835615937114, 'eval_aspect_f1': 0.5254813136531838, 'eval_opinion_f1': 0.48636859318871906, 'eval_pair_ta_f1': 0.33099297888679236, 'eval_pair_to_f1': 0.3055276381409419, 'eval_pair_ao_f1': 0.3245967741434959, 'eval_quad_micro_f1': 0.19293218715161944, 'eval_runtime': 8.8512, 'eval_samples_per_second': 32.764, 'eval_steps_per_second': 16.382, 'epoch': 8.0}
{'loss': 0.1513, 'learning_rate': 2.4115523465703976e-05, 'epoch': 8.05}
{'loss': 0.1359, 'learning_rate': 2.2310469314079425e-05, 'epoch': 8.94}
{'eval_loss': 0.2927750051021576, 'eval_target_f1': 0.5804676753281284, 'eval_aspect_f1': 0.5249709639452341, 'eval_opinion_f1': 0.4849833147441465, 'eval_pair_ta_f1': 0.32823649332415594, 'eval_pair_to_f1': 0.3108384457578278, 'eval_pair_ao_f1': 0.3268242548318063, 'eval_quad_micro_f1': 0.1966893865130186, 'eval_runtime': 8.8726, 'eval_samples_per_second': 32.685, 'eval_steps_per_second': 16.342, 'epoch': 9.0}
{'loss': 0.1281, 'learning_rate': 2.0505415162454874e-05, 'epoch': 9.84}
{'eval_loss': 0.29930731654167175, 'eval_target_f1': 0.5903448275360602, 'eval_aspect_f1': 0.5389908256379522, 'eval_opinion_f1': 0.49234135662385886, 'eval_pair_ta_f1': 0.3534136545684545, 'eval_pair_to_f1': 0.3101711983387138, 'eval_pair_ao_f1': 0.34949494944489545, 'eval_quad_micro_f1': 0.20478468894531462, 'eval_runtime': 8.8807, 'eval_samples_per_second': 32.655, 'eval_steps_per_second': 16.328, 'epoch': 10.0}
{'loss': 0.1216, 'learning_rate': 1.8700361010830327e-05, 'epoch': 10.73}
{'eval_loss': 0.30461862683296204, 'eval_target_f1': 0.5733882029677025, 'eval_aspect_f1': 0.5294797687360048, 'eval_opinion_f1': 0.46954595786797404, 'eval_pair_ta_f1': 0.35829959509170983, 'eval_pair_to_f1': 0.30831642997030373, 'eval_pair_ao_f1': 0.3374233128334136, 'eval_quad_micro_f1': 0.20656370651385747, 'eval_runtime': 8.8613, 'eval_samples_per_second': 32.727, 'eval_steps_per_second': 16.363, 'epoch': 11.0}
{'loss': 0.1205, 'learning_rate': 1.6895306859205777e-05, 'epoch': 11.63}
{'eval_loss': 0.2945067286491394, 'eval_target_f1': 0.5710344827084793, 'eval_aspect_f1': 0.5173210161161623, 'eval_opinion_f1': 0.48720800884870674, 'eval_pair_ta_f1': 0.33096446695510423, 'eval_pair_to_f1': 0.31460674152308465, 'eval_pair_ao_f1': 0.3347022586768916, 'eval_quad_micro_f1': 0.19825072881317327, 'eval_runtime': 8.8625, 'eval_samples_per_second': 32.722, 'eval_steps_per_second': 16.361, 'epoch': 12.0}
{'loss': 0.1129, 'learning_rate': 1.5090252707581228e-05, 'epoch': 12.52}
{'eval_loss': 0.3026190400123596, 'eval_target_f1': 0.6030013642063567, 'eval_aspect_f1': 0.5315005726875676, 'eval_opinion_f1': 0.4901098900597943, 'eval_pair_ta_f1': 0.34406438626789665, 'eval_pair_to_f1': 0.3195146612240195, 'eval_pair_ao_f1': 0.3414634145841067, 'eval_quad_micro_f1': 0.2040423483620676, 'eval_runtime': 8.8855, 'eval_samples_per_second': 32.637, 'eval_steps_per_second': 16.319, 'epoch': 13.0}
{'loss': 0.1044, 'learning_rate': 1.3285198555956679e-05, 'epoch': 13.42}
{'eval_loss': 0.304652601480484, 'eval_target_f1': 0.5928961748132633, 'eval_aspect_f1': 0.5622857142355911, 'eval_opinion_f1': 0.4841182912971082, 'eval_pair_ta_f1': 0.3581488933101474, 'eval_pair_to_f1': 0.3174924165324123, 'eval_pair_ao_f1': 0.34517766492457647, 'eval_quad_micro_f1': 0.2173076922578169, 'eval_runtime': 8.8082, 'eval_samples_per_second': 32.924, 'eval_steps_per_second': 16.462, 'epoch': 14.0}
{'loss': 0.1056, 'learning_rate': 1.1480144404332132e-05, 'epoch': 14.31}
{'eval_loss': 0.3082675039768219, 'eval_target_f1': 0.6109589040594521, 'eval_aspect_f1': 0.544202066540003, 'eval_opinion_f1': 0.4972375690106856, 'eval_pair_ta_f1': 0.35829959509170983, 'eval_pair_to_f1': 0.3313008129581538, 'eval_pair_ao_f1': 0.3496932514837178, 'eval_quad_micro_f1': 0.2166344293505486, 'eval_runtime': 8.8181, 'eval_samples_per_second': 32.887, 'eval_steps_per_second': 16.443, 'epoch': 15.0}
{'loss': 0.1003, 'learning_rate': 9.675090252707581e-06, 'epoch': 15.21}
{'eval_loss': 0.3161661624908447, 'eval_target_f1': 0.5997248967861699, 'eval_aspect_f1': 0.5506912441895048, 'eval_opinion_f1': 0.5071982280783754, 'eval_pair_ta_f1': 0.35532994918860183, 'eval_pair_to_f1': 0.3367346938275924, 'eval_pair_ao_f1': 0.3514902363323152, 'eval_quad_micro_f1': 0.21595330734319937, 'eval_runtime': 8.8343, 'eval_samples_per_second': 32.827, 'eval_steps_per_second': 16.413, 'epoch': 16.0}
{'loss': 0.0962, 'learning_rate': 7.870036101083033e-06, 'epoch': 16.1}
{'loss': 0.093, 'learning_rate': 6.064981949458484e-06, 'epoch': 16.99}
{'eval_loss': 0.3212610185146332, 'eval_target_f1': 0.6101231189649148, 'eval_aspect_f1': 0.5425287355820603, 'eval_opinion_f1': 0.4934210525814794, 'eval_pair_ta_f1': 0.35448136953710047, 'eval_pair_to_f1': 0.3238866396261218, 'eval_pair_ao_f1': 0.34658511717728624, 'eval_quad_micro_f1': 0.20616570322566852, 'eval_runtime': 8.8367, 'eval_samples_per_second': 32.818, 'eval_steps_per_second': 16.409, 'epoch': 17.0}
{'loss': 0.0904, 'learning_rate': 4.259927797833936e-06, 'epoch': 17.89}
{'eval_loss': 0.32145148515701294, 'eval_target_f1': 0.6092896174362096, 'eval_aspect_f1': 0.5448275861567724, 'eval_opinion_f1': 0.4879120878619925, 'eval_pair_ta_f1': 0.3572149343596847, 'eval_pair_to_f1': 0.3221884497980371, 'eval_pair_ao_f1': 0.3448979591336439, 'eval_quad_micro_f1': 0.20849420844435906, 'eval_runtime': 8.8475, 'eval_samples_per_second': 32.777, 'eval_steps_per_second': 16.389, 'epoch': 18.0}
{'loss': 0.0909, 'learning_rate': 2.4548736462093864e-06, 'epoch': 18.78}
{'eval_loss': 0.32269975543022156, 'eval_target_f1': 0.6101231189649148, 'eval_aspect_f1': 0.5419058552885683, 'eval_opinion_f1': 0.49504950490039973, 'eval_pair_ta_f1': 0.35555555550555734, 'eval_pair_to_f1': 0.3286004056295288, 'eval_pair_ao_f1': 0.34933605715119825, 'eval_quad_micro_f1': 0.21062801927382796, 'eval_runtime': 8.8492, 'eval_samples_per_second': 32.771, 'eval_steps_per_second': 16.386, 'epoch': 19.0}
{'loss': 0.0882, 'learning_rate': 6.498194945848375e-07, 'epoch': 19.68}
{'eval_loss': 0.3236048221588135, 'eval_target_f1': 0.6073871408527405, 'eval_aspect_f1': 0.5366972476563013, 'eval_opinion_f1': 0.4977973567780998, 'eval_pair_ta_f1': 0.34747474742475093, 'eval_pair_to_f1': 0.3286004056295288, 'eval_pair_ao_f1': 0.3472931562318932, 'eval_quad_micro_f1': 0.20676328497431182, 'eval_runtime': 8.854, 'eval_samples_per_second': 32.754, 'eval_steps_per_second': 16.377, 'epoch': 20.0}
{'train_runtime': 2227.8463, 'train_samples_per_second': 20.073, 'train_steps_per_second': 5.018, 'train_loss': 0.26315246779931467, 'epoch': 20.0}

以 eval_quad_micro_f1 來挑選 best model 的話，第14個 epoch 最佳。

{'eval_loss': 0.304652601480484, 'eval_target_f1': 0.5928961748132633, 'eval_aspect_f1': 0.5622857142355911, 'eval_opinion_f1': 0.4841182912971082, 'eval_pair_ta_f1': 0.3581488933101474, 'eval_pair_to_f1': 0.3174924165324123, 'eval_pair_ao_f1': 0.34517766492457647, 'eval_quad_micro_f1': 0.2173076922578169, 'eval_runtime': 8.8082, 'eval_samples_per_second': 32.924, 'eval_steps_per_second': 16.462, 'epoch': 14.0}

Tensorboard