Esri / OptimizeRasters

OptimizeRasters is a set of tools for converting raster data to optimized Tiled TIF or MRF files, moving data to cloud storage, and creating Raster Proxies.
Apache License 2.0
127 stars 50 forks source link

Hardcoded use of S3 Virtual addressing/hosting #135

Closed mbeaucha closed 2 years ago

mbeaucha commented 3 years ago

Lines 337 and 3285 of OptimizeRasters.py hardcode the use of virtual addressing for S3. This breaks compatibility with certain regions and/or third-party S3 API implementations like MinIO. The code should be updated to either respect the value of 'addressing_style' in the ~/.aws/config file or the value of AWS_VIRTUAL_HOSTING, which is used by the embedded GDAL package to configure the same setting, or both. For completeness this should be a setting in the OptimizeRasters.xml file as well and an optional command line argument.

abhiataero commented 3 years ago

@mbeaucha optimize raster support the addressing style from /.aws/config so in that case it should just work. in the template files we refer it using the profile. The profile is defined in the .aws/config

https://github.com/Esri/OptimizeRasters/issues/97#issuecomment-881205958

Chamlika commented 3 years ago

@mbeaucha Hi, the addressing style was forced into 'virtual style' because the 'path style' type is no longer recommended by AWS and could be soon made obsolete if not already for the older clients. https://github.com/Esri/OptimizeRasters/issues/97#issuecomment-881205958. However, we'll look forward to take your recommendation to derive the path style using the ENV/config if set, else default to 'virtual' if omitted. Thanks!

abhiataero commented 2 years ago

@Chamlika has this been taken care of

Chamlika commented 2 years ago

@Chamlika has this been taken care of

@abhiataero Will look forward to include in the next release.

Chamlika commented 2 years ago

@mbeaucha Let us know if the fix works for you, thanks!

For more info on the fixes, https://github.com/Esri/OptimizeRasters/commit/25c3ae7281728666e56779ff08b63981401cd999