search

BCDevOps / OpenShift4-Migration

Scripts and info for Ministry teams migration from OpenShift 3.11 to 4.x
Apache License 2.0
3 stars 0 forks source link

My app stores large amounts of data, what storage options are available in OCP 4? #59

Closed mitovskaol closed 2 years ago

mitovskaol commented 3 years ago

For the applications that need to store large amounts of data - beyond the storage allocation included in the medium size quota as defined here - we recommend using the S3-compliant on-prem Object Storage Service offered by OCIO's Hosting Branch for storing unstructured data such as images, pdfs and other types of files. The NetApp storage offered on the OCP4 Platform is not suitable for storing large amounts of unstructured data. The NetApp storage should only be used for structured data such as databases that require high-I/O workloads.

Please contact your Ministry IMB to get access to the Ministry service account that controls access to the .objectstrorage.gov.bc.ca namespace in the S3 storage where a bucket for your app will be provisioned.

Object Storage Service is highly fault tolerant using erasure coding within the cluster, as well as asynchronous replication and automatic failover to the Calgary data center The solution also allows offers configurable object versioning, allowing for file recovery. Due to the high fault tolerance, versioning, and redundancy, most teams do not implement additional backups. A recommended design pattern would be to replace the app's Minio storage with the direct integration with the S3 API within the Object Storage Enterprise Services.

NRM has had significant experience with the Object Storage service and kindly shared their collection of tips and tricks and best practices for how to work with the Object Storage Enterprise Service, with the Platform Community. 🙏

Our own Artifactory service in the Silver cluster leverages the Object Storage Service behind the scene as image storage (ping @caggles if you have any question about the Artifactory setup).

You can connect with other teams that are working with the Object Storage Service in the #object-storage channel in RocketChat!

ccocker-hub commented 3 years ago

Thanks for this - if you are working in the NRS Namespace, please follow the 'How to Request Object Storage' instructions on the page through the NRM Business Service Desk.

jefkel commented 3 years ago

Cross posting from @BcGovNeal for an S3 mirroring walkthrough: https://github.com/BCDevOps/OpenShift4-Migration/issues/9#issuecomment-723287484

I have created a very simple Docker image layered on top of the official Minio Client docker image to assist Minio data migration between clusters.

This image can be run on OpenShift to enable cluster to cluster data migration.

The documentation ca be found in my repo at https://github.com/BcGovNeal/minio-client