De-duplicate data on Archive

[sg-s]

• [ ] run rmlint
• [ ] verify that only duplicated files will be deleted
• [ ] Run a small part of the script to check that only duplicates and not originals are deleted
• [ ] Run it on the whole dataset

[sg-s]

bump?

[cosmojg]

@sg-s I did this already, but I'm going to do it again once I've made sure everything that needs to be backed up is at least somewhere on crab (for example, there's still some stuff sitting on the fileshare from when crab was having an existential crisis). Let's keep this issue open for now.

[klwadland]

In addition to de-duplicating the data on the Archive we also need to dig through and delete unnecessary items (pictures of people's cat, dog, significant other, vacation, etc.; financial documents; resumes; pdfs of interesting papers; c:\ drive contents/programs; and so on). Is there a way to do automate or semi-automate this so someone doesn't have to review the whole Archive file by file?

[cosmojg]

@Liaro0903 Katelyn said she'll ask Steve about letting one of us go down there and help out with expanding the Archive. In the meantime, we should work on finding or writing a program to speed up the process of sorting through everything that's on there and deleting all the junk.

[klwadland]

Sent an email to Steve and am waiting to hear what he says.