Free Up Space on Isilon Storage

escowles commented 5 years ago

~~Blocked by #2556~~

[x] Identify and stage content on Isilon that can be archived to cloud storage (PUDL, Archives, and Studio files that have already been ingested)
[x] Transfer files to cloud storage (possibly using Snowball/Google Transfer Appliance)
[x] Verify that content is available
[x] Remove content from the Isilon

escowles commented 5 years ago

Decision: Use Glacier for this, since it's the cheapest

escowles commented 5 years ago

I have begun the process of cataloging what's on disk in the /pudl (86TB) and /archives (66TB) shares.

For /archives (66TB):

I have created a spreadsheet listing the collections and links to Figgy to review items in each collection
I've discussed review with Faith

For /pudl (86TB):

I have listed each directory and created a list of directories and how many directories and files they contain
I am in the process of querying Figgy to see if each object exists, if it's complete, and how many members it has
When that is complete, I will check each PUDL object against the Figgy content, and create a list of which PUDL collections appear to have all items ingested, and create a spreadsheet for review

Once a collection is signed off as complete, I plan to archive each one in a single Zip file, and transfer to Amazon Glacier.

escowles commented 4 years ago

Done with this for now — will revisit as more content is reviewed and cleared for archiving.

pulibrary / figgy