Closed thoppe closed 3 years ago
Data has already been pulled. Quality is generally high, though the charts, figures, and footers give some unusable sections. I've added a quick KL filter over bigrams to remove most of the poor quality text ... this helps a lot and reduces the total text from 900 MB to 800 MB.
This is complete, and the reproduction repo is here:
https://github.com/thoppe/The-Pile-CoastalZoneInformation
and link to the data is in the repo and can be found here (temporrary):
https://drive.google.com/file/d/1qjZZTqS-m63TMKBYB1eNRc5Bh4W--SYQ/view?usp=sharing
@StellaAthena feel free to close this once it has been pinned to the board and assign me.
Removed from the Pile by @leogao2 due to low quality. Despite our best efforts, horrible OCR made it unusable. @thoppe believes applying modern OCR to the original documents instead of using the processed data may allow improvements, but we are deferring that for now.
Language: English Date ranges: 1951-1998 Size: ~ 5,000 long form documents Text size: 800 MB
https://www.govinfo.gov/app/collection/czic
Sample text: