earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine
https://icechunk.io
Apache License 2.0
295 stars 18 forks source link

Object tagging #397

Open jhamman opened 6 days ago

jhamman commented 6 days ago

Most object stores allow writing object tags (or custom metadata) at the object level. Icechunk users could utilize object tags to control intelligent tiering behavior. For example, if all Icechunk metadata objects included the tag IcechunkType=Metadata, then this could be used to when defining s3 bucket settings to avoid moving metadata objects to lower storage tiers. There are likely other metadata tags that would be useful as well.