Removed unzipped files from `data/raw` after processing in Scibite ingest

Knowledge-Graph-Hub / kg-covid-19

An instance of KG Hub to produce a knowledge graph for COVID-19 response.

BSD 3-Clause "New" or "Revised" License

79 stars 26 forks source link

Closed justaddcoffee closed 4 years ago

justaddcoffee commented 4 years ago

Refactor Scibite ingest slightly so that unzipped files are removed from data/raw after processing.

Currently raw is being uploaded to S3 for each of our builds, and it's taking >1.5h, I think in part because of these unzipped files, see here