Open scottyhq opened 8 months ago
Good point. The S3 tags are required for plugins being run by HyP3; HyP3 uses them to distinguish which files uploaded by the plugin are data files, browse images, and log files. The S3 tags could be made optional when running a plugin outside of HyP3 as described in https://github.com/ASFHyP3/hyp3-isce2/issues/175 . It's worth mulling, and we'd be open to a PR, but I doubt it's a change we'll invest the time to make ourselves.
Separately, Express One Zone buckets are specifically optimized for workloads with high request volumes and low latency requirements. Per-request costs are half the cost of S3 Standard, but storage costs are seven times more expensive. I'd recommend against Express One Zone in favor of S3 Intelligent Tiering for most HyP3-adjacent workloads.
The bug
We were trying to upload outputs from hyp3-isce2 (0.9.2) running locally to S3 by using
--bucket
and--bucket-prefix
workflow arguments. It appears the latest S3 Express One Zone buckets (cheaper & faster supposedly but not redundant) don't support object tags, leading to the following traceback:Workaround options
I can see the usefulness of tags, but it would be nice to enable this bucket option, maybe with a conditional statement? Not sure how to check for this new bucket type, but could look into it if that seems like the best path forward?
Additional context
https://github.com/ASFHyP3/hyp3-isce2/issues/175#issuecomment-1888029892