search

EleutherAI / the-pile

MIT License
1.46k stars 126 forks source link

Coastal Zone Information Center Collection #44

Closed thoppe closed 3 years ago

thoppe commented 3 years ago

Language: English Date ranges: 1951-1998 Size: ~ 5,000 long form documents Text size: 800 MB

https://www.govinfo.gov/app/collection/czic

The Coastal Zone Information Center (CZIC) collection on this site provides access to nearly 5,000 coastal related documents that GPO received from the National Oceanic and Atmospheric Administration (NOAA) Central Library.

The collection provides almost 30 years of data and information crucial to the understanding of U.S. coastal management and NOAA's mission to sustain healthy coasts. These documents were originally submitted to the NOAA Office of Ocean and Coastal Resource Management (OCRM) by state coastal zone management programs in accordance with the Coastal Zone Management Act (CZMA) of 1972. These historic documents were provided to GPO by the content originator, and digitized for public use. For optimal viewing and for the original look and feel of the original documents, view the PDF versions.

Sample text:

       Begins with an analysis of environmental and cultural dynamics affecting
       prehistoric resources. An evaluation of the resources in each county follows,
       and an examination of the different stresses on prehistoric resources (from
       shore erosion, residential development, wetlands destruction, etc.). The
       management strategy discusses current programs, land use planning at dif-
       ferent levels of government, and a county by county approach. Concludes
       with an appendix on principal existing legislation which concerns prehistoric
       resources.
thoppe commented 3 years ago

Data has already been pulled. Quality is generally high, though the charts, figures, and footers give some unusable sections. I've added a quick KL filter over bigrams to remove most of the poor quality text ... this helps a lot and reduces the total text from 900 MB to 800 MB.

thoppe commented 3 years ago

This is complete, and the reproduction repo is here:

https://github.com/thoppe/The-Pile-CoastalZoneInformation

and link to the data is in the repo and can be found here (temporrary):

https://drive.google.com/file/d/1qjZZTqS-m63TMKBYB1eNRc5Bh4W--SYQ/view?usp=sharing

thoppe commented 3 years ago

@StellaAthena feel free to close this once it has been pinned to the board and assign me.

StellaAthena commented 3 years ago

Removed from the Pile by @leogao2 due to low quality. Despite our best efforts, horrible OCR made it unusable. @thoppe believes applying modern OCR to the original documents instead of using the processed data may allow improvements, but we are deferring that for now.