Closed Jehoshaph closed 1 year ago
Working on the cleanup and de-duplicate scripts
@Jehoshaph Might be better to just directly use the existing usenet dump that you linked or this uncleaned version they mention: https://aws.amazon.com/datasets/the-westburylab-usenet-corpus/.
Is there any other dedup/cleanup steps you want to run?
A couple of potential issues with that dataset. They still haven't responded to my request for access to the dataset.
Is there any other dedup/cleanup steps you want to run?
The only additional step is excluding messages with no replies.
@Jehoshaph
ah okay, thanks for clearing that up. Yeah in that case, it makes sense to do it ourselves
UseNet
Dataset URL -
UsenetArchives andInternetArchiveDoes the dataset exists in a scraped format ? No
Description
Procedure
Tag code blocks