SDSC S3 Storage

[dloy]

SDSC is in the process of replacing Minio S3 with a native S3. Minio is an S3 interface that runs on top of a file system. As I understand the new SDSC S3 runs directly on the lower data storage.

An initial use of this new S3 is the use of a temporary backup for content being repaired.

To use this new storage requires:

• [x] bucket allocation at SDSC for the content we need
• [x] obtain the credentials: endpoint, accesskey, secretkey, endpoint
• [x] create mrt-cloud configurations to support the new content for node table
• [ ] create mrt-cloud configurations to support the new content for yaml
• [x] modify NodeIO to handle the new service type (sdsc-s3) - serviceType is used for customizing to a common java aws sdk in configuration
• [ ] set up SSM credentials for new storage node (stg & prd)
• [ ] create inv db node values (stg & prd)

[dloy]

Part of this Issue will include the removal of Dryad node handling from cloud configurations.

• nodes tables
• yaml table

[elopatin-uc3]

Marking as complete. Separate ticket filed to track SDSC-provided migration to new S3 native API: #1701