search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.92k stars 2k forks source link

minio s3 cannot connect #8709

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

hi, all,

Thanks for this great project! I love it!

Just report the s3/minio issue(or I miss something), I follow the doc here: [http://docs.h2o.ai/h2o/latest-stable/h2o-docs/cloud-integration/ec2-and-s3.html#minio-instance|http://docs.h2o.ai/h2o/latest-stable/h2o-docs/cloud-integration/ec2-and-s3.html#minio-instance]

and I use minio official [MinIO playground]([https://docs.min.io/docs/s3cmd-with-minio|https://docs.min.io/docs/s3cmd-with-minio]):


# Setup endpoint
host_base = play.min.io:9000
host_bucket = play.min.io:9000
bucket_location = us-east-1
use_https = True

# Setup access keys
access_key =  Q3AM3UQ867SPQQA43P2F
secret_key = zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG

# Enable S3 v4 signature APIs
signature_v2 = False

and create [a file there]([https://play.min.io:9000/minio/h2o/sub.csv).|https://play.min.io:9000/minio/h2o/sub.csv).]

I create a core-site.xml like this:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>

<property>
<name>Dsys.ai.h2o.persist.s3.endPoint</name>
<value>play.min.io:9000</value>
</property>
<property>
<name>Dsys.ai.h2o.persist.s3.enable.path.style</name>
<value>true</value>
</property>
<property>
<name>Daws.AccessKeyId</name>
<value>Q3AM3UQ867SPQQA43P2F</value>
</property>

<property>
<name>Daws.SecretAccessKey</name>
<value>zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG</value>
</property>

</configuration>

and I run h2o like this: java -jar h2o.jar -hdfs_config core-site.xml

In flow ui, I do this: importFiles \[ "s3://Q3AM3UQ867SPQQA43P2F:zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG@h2o/sub.csv" ]

In flow ui, it complains:

Error evaluating cell
Error calling GET [http://192.168.0.189:54321/3/ImportFiles?path=s3%3A%2F%2FQ3AM3UQ867SPQQA43P2F%3Azuf%2BtfteSlswRu7BJ86wekitnifILbZam1KYY3TG%40h2o%2Fsub.csv|http://192.168.0.189:54321/3/ImportFiles?path=s3%253A%252F%252FQ3AM3UQ867SPQQA43P2F%253Azuf%252BtfteSlswRu7BJ86wekitnifILbZam1KYY3TG%2540h2o%252Fsub.csv]

ERROR MESSAGE: Unable to load AWS credentials from any provider in the chain

the console log says:

10-01 08:20:43.630 192.168.0.189:54321   13139  #25991-26 INFO: GET /3/ImportFiles, parms: \{path=s3://Q3AM3UQ867SPQQA43P2F:zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG@h2o/sub.csv}
10-01 08:20:43.631 192.168.0.189:54321   13139  #25991-26 INFO: ImportS3 processing (s3://Q3AM3UQ867SPQQA43P2F:zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG@h2o/sub.csv)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3820)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3778)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:610)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:588)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.persist.PersistS3.importFiles(PersistS3.java:205)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.persist.PersistManager.importFiles(PersistManager.java:379)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.api.ImportFilesHandler.importFiles(ImportFilesHandler.java:25)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.api.Handler.handle(Handler.java:60)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.api.RequestServer.serve(RequestServer.java:462)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.api.RequestServer.doGeneric(RequestServer.java:295)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.api.RequestServer.doGet(RequestServer.java:219)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at water.webserver.jetty8.Jetty8ServerAdapter$LoginHandler.handle(Jetty8ServerAdapter.java:119)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.Server.handle(Server.java:370)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
10-01 08:20:45.960 192.168.0.189:54321   13139  #25991-26 ERRR:         at java.base/java.lang.Thread.run(Thread.java:834)
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: Caught exception: 
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: 
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: ERROR MESSAGE:
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: 
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: Unable to load AWS credentials from any provider in the chain
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: 
10-01 08:20:45.961 192.168.0.189:54321   13139  #25991-26 ERRR: ; Stacktrace: \[com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain,     com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131),     com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3820),     com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3778),     com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:610),     com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:588),     water.persist.PersistS3.importFiles(PersistS3.java:205),     water.persist.PersistManager.importFiles(PersistManager.java:379),     water.api.ImportFilesHandler.importFiles(ImportFilesHandler.java:25),     java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method),     java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),     java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),     java.base/java.lang.reflect.Method.invoke(Method.java:566),     water.api.Handler.handle(Handler.java:60),     water.api.RequestServer.serve(RequestServer.java:462),     water.api.RequestServer.doGeneric(RequestServer.java:295),     water.api.RequestServer.doGet(RequestServer.java:219),     javax.servlet.http.HttpServlet.service(HttpServlet.java:735),     javax.servlet.http.HttpServlet.service(HttpServlet.java:848),     org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684),     org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501),     org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086),     org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:427),     org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020),     org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135),     org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154),     org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116),     water.webserver.jetty8.Jetty8ServerAdapter$LoginHandler.handle(Jetty8ServerAdapter.java:119),     org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154),     org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116),     org.eclipse.jetty.server.Server.handle(Server.java:370),     org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494),     org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53),     org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:973),     org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1035),     org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:641),     org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:231),     org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72),     org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264),     org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608),     org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543),     java.base/java.lang.Thread.run(Thread.java:834)];parms=\{path=s3://Q3AM3UQ867SPQQA43P2F:zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG@h2o/sub.csv}
h2o-ops commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-6927 Assignee: New H2O Bugs Reporter: Yuxi Luo State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A