This module provides function called tfidf_subset_sampler to sample subsets based on TF-IDF similarity and save the filterd train data and return the indices of orignial train set. It is designed for benchmark nanoGPT.
1. Motivation and Context
To solve the problem in issue #73.
2. Summary of the change
Add a new file called tfidf_sampler.py under model_utils. The callable function tfidf_subset_sampler is included in it
3. What tests have been added/updated for the change?
Description
This module provides function called tfidf_subset_sampler to sample subsets based on TF-IDF similarity and save the filterd train data and return the indices of orignial train set. It is designed for benchmark nanoGPT.
1. Motivation and Context
To solve the problem in issue #73.
2. Summary of the change
Add a new file called tfidf_sampler.py under model_utils. The callable function tfidf_subset_sampler is included in it
3. What tests have been added/updated for the change?