QD Synthetic Data

This repository serves as the main organizational tool for the survey paper "A Survey of Methods for Generating Quality and Diverse Synthetic Data with LLMs". We are collecting papers as Github issues with the tag Paper. To add a new paper, first check that it is not present, then fill out the new paper issue template here. To close the issue you (or someone else) can make a PR containing a report on the paper using the provided format here. You can find a roadmap for the project on this Github projects board. Weekly meeting notes and recordings are housed here.

Project Description

The aim of this project is to catalog the many current ad hoc methods for synthetic data generation via LLMs with a focus on understanding their impact on two metrics: dataset quality and dataset diversity. Ideally, this can be done under a single conceptual framework. An important sub-question we will need to discuss is how to appropriately define these metrics, in particular dataset diversity.

Overall, this will roughly consist of three stages:

1. Collection: finding as many papers as we can pertaining to synthetic data generation and measures of and techniques improving quality or diversity for LLMs.
2. Synthesis: writing a survey of our findings by organizing methods into a single conceptual framework. We may also want to do some benchmarking.
3. Next steps: identifying promising research directions as recommendations to the broader community

Questions

Some important questions we will want to think about addressing:

• How to define quality?
• How to define diversity?
• The impact of the model
  • model size
  • pretraining data
  • fine-tuning data
• The impact of the sampling methodology
  • Type of prompt
  • Sampling algorithm
• Impact of the task domain

Meeting Time

Meetings are at 5:30 PM EST on Thursdays. Email alexdahoas@gmail.com or DM Alex Havrilla on discord for access.

Links

• Meeting Notes: Link
• Paper: Link
• Project board: Link