Open josh-chamberlain opened 5 months ago
- [ ] check for new URLs in our database which aren't already in the training-urls dataset via the API
Where will these new raw URLs be hosted? At the moment, my common_crawler pr #45 simply stores new URLs in the repository, which obviously isn't a sustainable long-term option.
@maxachis this is a good point—in general, I think we should use hugging face datasets.
I added detail to this issue: https://github.com/Police-Data-Accessibility-Project/data-source-identification/issues/40
@josh-chamberlain To make sure I fully understand the workflow:
training-urls
datasettraining-urls
dataset pulled in 2. Update the last_updated
property of all new entries (or of the entire dataset?)training-urls
datasetAdditionally, when we are talking about training-urls
dataset, does this dataset currently exist? In PDAP's Hugging Face, I do not currently see a dataset named training-urls
:
@maxachis
5. we don't need to insert into LabelStudio—to be clear, we are checking LabelStudio for newly labeled URLs which aren't already in our training data.
training-urls
doesn't currently exist, creating the dataset + strategy for managing batches of URLs within it.
I updated the readme for this repo and tweaked this issue slightly—I think using Hugging Face as a database for un-labeled URLs is not needed. We can track batches by ID in github, but we don't need to put them in hugging face before they're labeled. Hopefully this is much simpler.
Context
Now that we've done it a few times, let's be systematic about how we update the base training dataset in Hugging Face.
Requirements
training-urls
as the canonical dataset in Hugging Face (HF), which we will use Pull Requests in to maintain. We will likely remove the other ones.Docs