remove unnecessary ".DS_store" files from wasabi bucket

[only1chunts]

User story

As a curator I want remove any Mac specific system files called ".DS_store" from the Wasabi bucket So that we're not paying to host unnecessary small data files.

Acceptance criteria

Given new dataset files have been added to the dropboxes
When we make them public
Then they don't contain hidden system files (like ".DS_store")

Given files have been uploaded to Wasabi
When they contains hidden system files Then they should be remove from the Wasabi bucket

Additional Info

.DS_store files are created on the fly by the mac OS, they are useless in any other environment, and the mac OS regenerates them anyway so they are also not required to be downloaded even for mac users, so they serve no purpose in GigaDB datasets.

Implementation notes

• wrapper script around find/grep/awk curators can run to detect those files in the dropboxes
• set global alias for e.g: ls to be ls -a

Product Backlog Item Ready Checklist

• [ ] Business value is clearly articulated
• [ ] Item is understood enough by the IT team so it can make an informed decision as to whether it can complete this item
• [ ] Dependencies are identified and no external dependencies would block this item from being completed
• [ ] At the time of the scheduled sprint, the IT team has the appropriate composition to complete this item
• [ ] This item is estimated and small enough to comfortably be completed in one sprint
• [ ] Acceptance criteria are clear and testable
• [ ] Performance criteria, if any, are defined and testable
• [ ] The Scrum team understands how to demonstrate this item at the sprint review

Product Backlog Item Done Checklist

• [ ] Item(s) in increment pass all Acceptance Criteria
• [ ] Code is refactored to best practices and coding standards
• [ ] Documentation is updated as needed
• [ ] Data security has not been compromised (with particular reference to the personal information we hold in GigaDB)
• [ ] No deviation from the team technology stack and software architecture has been introduced
• [ ] The product is in a releasable state (i.e. the increment has not broken anything)

[pli888]

There might be other types of hidden files buried in datasets. I'm sure I've seen .swp files in datasets before which can cause problems for people if they download dataset files and then open them using Vim text editor.

[only1chunts]

this functionality may be covered by #1869