search

minio / minio

MinIO is a high-performance, S3 compatible object store, open sourced under GNU AGPLv3 license.
https://min.io/download?license=agpl&platform=linux
GNU Affero General Public License v3.0
48.1k stars 5.5k forks source link

Glacier Tier Support with External S3 Storage #19823

Closed gaohlf closed 5 months ago

gaohlf commented 5 months ago

Is your feature request related to a problem? Please describe. Currently, MinIO supports S3 as its glacier tier. However, the format in which the data is stored in the external S3 storage is only accessible by MinIO. Upon reviewing the implementation, it appears that files are organized into shard objects and the object names are reorganized using the hash value of the original file. This is quite inconvenient. If a user's MinIO frontend application is destroyed, it becomes very troublesome to re-establish connections. I believe if the glacier tier's organization method was the same as the original file storage method, it would save a lot of hassle.

Describe the solution you'd like Therefore, I propose adding a switch that users can enable when starting the MinIO cluster. If enabled, this switch would allow the glacier tier structure to be the same as the standard storage structure.

Describe alternatives you've considered None.

Additional context None.

harshavardhana commented 5 months ago

Transitioning to glacier must be set on the target bucket on AWS S3. We have no plans to directly upload to glacier.

ravindk89 commented 5 months ago

As per our docs

MinIO Object Transition supports cost-saving strategies around moving older or aged data to cost-optimized remote storage tiers, such as cloud storage or high-density HDD storage.

MinIO Object Transition does not provide backup and recovery functionality. You cannot use the remote tier as a recovery source in the event of data loss in MinIO.

It has nothing to do with glacier being the remote tier - which we do not even support as-is, because glacier requires rehydration to retrieve data, and our docs also warn against this.

All tiered data is stored w/ UUID, and is intended for access exclusively through the MinIO hot tier. This is not going to change.