Mentor-KD

Mentor-KD: Making Small Language Models Better Multi-step Reasoners

Hojae Lee, Junho Kim, SangKeun Lee (*: equal contributon)

This repository contains the code and datasets used for our study "Mentor-KD: Making Small Language Models Better Multi-step Reasoners" accepted to EMNLP 2024 (main). In this paper, we improve reasoning distillation by leveraging task-specific intermediate-sized mentor models which complements insufficient distillation signals from the teacher model.

👷‍♂️ Some code files will be refactored for better readability in the near future!

Overview

overview

Requirements

Required Tools

Python >= 3.10
torch >= 2.0.0
transformers >= 4.41.2
peft >= 0.4.0

Data

All data used in our study can be found in the data/ directory. As mentioned in Appendix A, we use CoT annotations as well as train-test dataset splits previously collected by MCC-KD[^mcckd], specifically for CommonsenseQA, GSM8K, ASDiv, SVAMP. For other datasets, we newly prompt gpt-3.5-turbo-instruct and collect CoT annotations, while following the train-test dataset splits from Fine-tune-CoT[^ftcot].

Please refer to each studies' GitHub repository for the original data: [MCC-KD] [Fine-tune-CoT]

Reproduction

We provide shell scripts to reproduce the results reported in our paper as below.

A task-specific intermediate-sized mentor model must be trained before conducting reasoning distillation to the student. We provide shell scripts for training the mentor model, using it for means of rationale augmentation, and finally for training the student models via our Mentor-KD framework.

Training the Mentor Model

bash shells/fine_tune_cot/run_fine_tune_cot.sh

Rationale Augmentation via the Mentor Model

bash shells/data/augment_train_data.sh

Reasoning Distillation (Training the Student Model)

bash shells/kd/run_mentor_kd.sh

Citation

@inproceedings{lee-etal-2024-mentor,
    title = "Mentor-{KD}: Making Small Language Models Better Multi-step Reasoners",
    author = "Lee, Hojae  and
      Kim, Junho  and
      Lee, SangKeun",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.977",
    pages = "17643--17658",
    abstract = "Large Language Models (LLMs) have displayed remarkable performances across various complex tasks by leveraging Chain-of-Thought (CoT) prompting. Recently, studies have proposed a Knowledge Distillation (KD) approach, reasoning distillation, which transfers such reasoning ability of LLMs through fine-tuning language models of multi-step rationales generated by LLM teachers. However, they have inadequately considered two challenges regarding insufficient distillation sets from the LLM teacher model, in terms of 1) data quality and 2) soft label provision. In this paper, we propose Mentor-KD, which effectively distills the multi-step reasoning capability of LLMs to smaller LMs while addressing the aforementioned challenges. Specifically, we exploit a mentor, intermediate-sized task-specific fine-tuned model, to augment additional CoT annotations and provide soft labels for the student model during reasoning distillation. We conduct extensive experiments and confirm Mentor-KD{'}s effectiveness across various models and complex reasoning tasks.",
}

Acknowledgements

We thank the authors of MCC-KD[^mcckd] and Fine-tune-CoT[^ftcot] for open-sourcing the data used for their respective works.

Point of Contact

For personal communication related to Mentor-KD, please contact Hojae Lee via 22leehojae@korea.ac.kr.

[^mcckd]: Chen et al., "MCC-KD: Multi-CoT Consistent Knowledge Distillation", Findings of EMNLP 2023. [^ftcot]: Ho et al., "Large Language Models are Reasoning Teachers", ACL 2023.

2hojae / mentor-kd

readme