SDSC data migration from Minio to native S3

[dloy]

Minio

Minio is an S3 API based service that preserves the content on a standard file system. SDSC is planning to move from using Minio to a "native S3 service". The native service directly writes content to block storage content below that is not file based.

Minio is able to mimic S3 by storing metadata content describing the files in a sidecar file that contains these properties.

Because Minio acts as an interface it has had a number of problems supporting the AWS S3 API - especially on some of the lesser used functions:

• get next key - hangs the system
• TransferManager activity that downloads multiple parts of large content as multiple threads typically fails with large content.

Getting off Minio is a good thing!

From Gavin:

From: David Loy <[David.Loy@ucop.edu](mailto:David.Loy@ucop.edu)>
Date: Wednesday, November 29, 2023 at 1:10 PM

...

Hi Gavin
...
Will our existing content be available on the new service or will there be some form of migration required?

David

---------
Hi David, 
...
Unfortunately, I think we may need to migrate the data. MinIO stores the object metadata on the file system in 
a sidecar file within a shadow directory structure. We can expose the existing buckets as the keys and object 
content would be the same, but they’d be missing the metadata. We may be able to re-ingest the data locally 
to speed up the process. It’s a challenge that’s worth thinking about early, but we have some time. 

Thanks,
Gavin

Amount of content

Currently we have ~572TB content saved at SDSC.

Problem

If SDSC relies on us to transfer this content from Minio to an S3 native service at SDSC we are looking at some excessive data movement costs with AWS on > 1/2 P data. Input content to AWS is not charged (according to one source). Content sent from within AWS to an outside location is charged.

Getting the specific cost for DataTransfer-Out-Bytes or REGIONNAME1-DataTransfer-Out-Bytes is difficult. One source has this as "Next 100 TB / Month | $0.084 per GB. So 572000 x $0.084" = $48,048. This would strictly be the output transfer costs and does not include:

• contractual discounts
• person/program costs for handling transfer
• duplication costs at SDSC if we are charged with both storages during transfer

Suggestion

Work with SDSC to find an onsite way to transfer content. "We may be able to re-ingest the data locally to speed up the process." This would be excellent and then leave us with the issues of dealing with:

• possible duplication costs
• down time for either syncing updates or cutting over

If SDSC cannot handle the "ingest" cost mentioned above, other possible solutions might include a temporary cloud service on their system for our code doing the transfer. Better if handled by SDSC.

Important that we start discussions with them about this migration.

[elopatin-uc3]

@elopatin-uc3 Will set up a meeting with SDSC to discuss.

[elopatin-uc3]

Meeting with SDSC on 1/11.

[terrywbrady]

Jan 11 meeting with SDSC

• [ ] Gavin will explore migration options to handle the replication for us. It is not clear how long the process will take.
• [ ] Gavin will grant David access to a DEV system to try things out
• [ ] David - define/document migration steps after the replication is complete
• [ ] David - test out the DEV environment
• [ ] Team - determine if we see value in splitting primary and secondary content into different SDSC nodes
• [ ] Team - determine Merritt operational status during the migration (pause replication, pause ingest?)

[elopatin-uc3]

Related: #1927