search

pugetsoundandvision / videotools

BSD 3-Clause "New" or "Revised" License
9 stars 1 forks source link

Option to --excludehiddenfiles from bags #18

Closed alavigne12 closed 5 years ago

alavigne12 commented 5 years ago

Our computers are generating tons of hidden files, which cause our bags to not verify. Ideally, we would have an option to use the 'exclude hidden files' option when the bag is created. [I think Andrew already did this but I can't remember]

privatezero commented 5 years ago

Yah - this is already in effect, see: https://github.com/pugetsoundandvision/videotools/blob/master/videoaip#L241

additionally, the syncing also is set to ignore hidden files (well, anything that starts with a .): https://github.com/pugetsoundandvision/videotools/blob/master/videoaip#L253

So those files must be creeping in during some other part of the workflow

alavigne12 commented 5 years ago

Alright. We also haven't made any new AIPS since you fixed it, so it should work, just haven't tested it yet.

privatezero commented 5 years ago

I mean, the exclude hidden files tag has been in effect since march, and seems to have been working as intended. If hidden files somehow are created at time of AIP creation, they now at least will not be transferred to your AIP sync location, but I have serious doubts that is the problem, so even with testing I don't think we are there yet.

I think that files are being generated after the AIPs are made and moved, which leaves us with the problem of them being present but not in the manifest. This messes up the lto manifest script, which uses bagit ruby instead of baggit java, because it doesn't have an option to ignore hidden files.

I think we will need to A: try to pin down what is going on in your workflows that cause so many resource forks to be generated, and B: add either a new script or an addition to the lto manifest script that erases hidden files.