Closed Jehoshaph closed 1 year ago
Hello, thanks for the PR. I just had a brief look, we want to inherit classes from this base class. https://github.com/CarperAI/Code-Pile/blob/working/codepile/dataset.py Please have a look.
Do we have an estimate for how big this dataset is?
Do we have an estimate for how big this dataset is?
Usenet Comp (link) is around 30 GB
Hey @Jehoshaph, how is the status of this PR?
Hey @Jehoshaph, how is the status of this PR?
It still needs some work, I'll be updating the PR soon.
Hey @Jehoshaph, how is the status of this PR?
@ncoop57 Updated! Ready for review.
@Jehoshaph could you fix the test/usenet-comp.parquet dummy data? I tried running the tests, but it failed since it is a folder rather than the parquet file. Also, could you merge the changes from working into your local branch.
@Jehoshaph could you fix the test/usenet-comp.parquet dummy data? I tried running the tests, but it failed since it is a folder rather than the parquet file. Also, could you merge the changes from working into your local branch.
@ncoop57 That is strange, the test works for me. I am using pyarrow to incrementally add to parquet since I am very memory constrained on my machine. Pyarrow parquet works on directories, and calling pyarrow.parquet.read_table('dir_name')
works the same as a file. I've moved the test file and made some updates, let me know if it still does not work, I will need to investigate this more.
Merged working into my local branch.
Works like a charm, thanks @Jehoshaph !
Adds UseNet download and processing scripts. Refer issue #16