instructlab / sdg

Python library for Synthetic Data Generation
https://pypi.org/project/instructlab-sdg/
Apache License 2.0
23 stars 35 forks source link

Allow documents to be loaded locally for Knowledge Injection #388

Open murthyrudra opened 1 week ago

murthyrudra commented 1 week ago

This feature allows users to specify the local folder path instead of github repo for knowledge injection. Specifically, the update modifies the validate_document function to search for files locally and ensure the documents follow the prescribed format.

This feature includes an update to the schema repo and the corresponding PR is here