Closed achen353 closed 2 years ago
@stephanieeechang I'm attaching the processed files here (because some go beyond the 100MB file size limit for GitHub and would require Github LFS).
https://drive.google.com/file/d/1sp8h1hSZRzJGtYaA0zzxXS6hZbGdfG_5/view?usp=sharing
These files are NOT FINALIZED. Due to time constraints, I'm just sharing this with you so you can better work on your part based on the JSON structure. See the Description of this PR to see why it's not finalized.
There are four JSON files and once you extract the zip file, you should put them in /datasets/billsum_extractive/
@andywang268 You should also be aware of this too.
Context
19
Summary
Relevant Technical Choices
multiprocessing
module does not allow nested pools to be created. A more sophisticated multiprocessing library is required.Sample Processed Data
Note
There are some parameters to be tuned after the merge of this PR. For example, we can see that most of the labels are
0
and it would affect the training. This JSON snippet is just for reference of what the structure looks like.