Modify convert_to_extractive.py

achen353 / TransformerSum

BERT-based extractive summarizer for long legal document using a divide-and-conquer approach

GNU General Public License v3.0

3 stars 0 forks source link

Closed achen353 closed 2 years ago

achen353 commented 2 years ago

Convert the billsum HuggingFace dataset into an abstractive dataset

TODO

[x] Fix any bug encountered
[x] Add code to split train split into train and validation (since billsum on Huggingface only has train, test, ca_test splits)
[x] Check if we need to remove extra white spaces and escaped characters like \n and \t