Aligning ML Models with Human Feedback
- What this repository contains:
- Establishing Supervised Model Baseline
- Gathering and Incorporating Human Feedback
- Training and Assessing the Final Model with Reinforcement Learning
What this repository contains:
This repository contains a collection of tutorials, best practices, and references for developers, data scientists, and machine learning professionals of all skill levels.
This repo may be of interest to you if:
- You've attended the PyData Berlin Talk by Nikolai Liubimov and Erin Mikail Staples
- You're interested in data-centric model development
- You're using Label Studio to improve your ML process and models.
If you have any questions about this repo, or need a hand:
- This repo is maintained by the Label Studio team.
- Reach out to the Label Studio Community Team or open an issue with any questions.
- Join the Label Studio Community Slack
- Check out more resources around Human in the Loop / RHLF in our Awesome List
Establishing Supervised Model Baseline
In this step, we collect labeled text data to train an initial Large Language Model (LLM), focusing on task-specific performance improvements. This stage involves gathering instructions and responses to adapt the base model to a broad range of tasks, enhancing its ability to generate accurate and contextually relevant responses. Typically this step involves:
- Selecting baseline Foundational Model (FM) that can perform fairly well on general tasks (like GPT2 or GPT-J)
- Generate dataset of pairs prompt input followed by response. You can manually label dataset or generate the data like it is provided in the example
- Peform supervised model finetuning.
Gathering and Incorporating Human Feedback
This stage involves collecting comparison data to establish human preferences for the responses generated by the supervised model. By ranking multiple responses based on quality, we can train a reward model that effectively captures human preferences. This reward model plays a crucial role in reinforcement learning, optimizing the performance of the fine-tuned foundational model.
Gathering Human Feedback Tutorial - This Jupyter Notebook tutorial will guide you through the process of collecting comparison data, establishing human preferences, and incorporating this feedback into the reward model training.
Training and Assessing the Final Model with Reinforcement Learning
The training stage is a challenging process, and the final model assessment is a critical component in evaluating your model's quality. It is essential to determine whether the model adheres to the provided instructions, avoids biases, and maintains a high standard of performance.