search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.94k stars 2k forks source link

Allows to specify an arbitrary S3 endpoint/region #10236

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

To support Minio S3 API, we need to setup an arbitrary S3 endpoint instead of default 's3.amazonaws.com`. Right now, it seems AWS SDK does not allow do that via system/env properties and the setup need to be performed during client initialization.

See code: https://gist.github.com/krishnasrinivas/edc62873bcef5396493f31f8d582fc2c

exalate-issue-sync[bot] commented 1 year ago

Michal Malohlava commented: For minio we also need path style support:

{{ Amazon S3 supports virtual-hosted-style and path-style access in all Regions. The path-style syntax, however, requires that you use the region-specific endpoint when attempting to access a bucket.

The default behaviour is to detect which access style to use based on the configured endpoint (an IP will result in path-style access) and the bucket being accessed (some buckets are not valid DNS names). Setting this flag will result in path-style access being used for all requests. }}

exalate-issue-sync[bot] commented 1 year ago

Bill Gallmeister commented: Hmm, PR in progress already...

exalate-issue-sync[bot] commented 1 year ago

Michal Malohlava commented: Scott is testing it - minor update in S3 client configuration. We will be able to point to non-AWS S3 .

h2o-ops commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-3321 Assignee: Michal Malohlava Reporter: Michal Malohlava State: Open Fix Version: N/A Attachments: N/A Development PRs: Available

Linked PRs from JIRA

https://github.com/h2oai/h2o-3/pull/164