Add wiki retrieval - Githubissues

VikParuchuri / textbook_quality

Generate textbook-quality synthetic LLM pretraining data

MIT License

461 stars 46 forks source link

Add wiki retrieval #12

Closed VikParuchuri closed 8 months ago

VikParuchuri commented 8 months ago

Add custom search retrieval
Remove some unnecessary async generators
Add rewrite pipeline