Alluxio / alluxio

Alluxio, data orchestration for analytics and machine learning in the cloud
https://www.alluxio.io
Apache License 2.0
6.73k stars 2.92k forks source link

Regarding the problem that calling S3 API cannot be tested #18615

Open lgy1027 opened 1 month ago

lgy1027 commented 1 month ago

Hello, I am currently using fluid and alluxio for cache acceleration, and set them in alluxioruntime. apiGateway: enabled: true , but when I request the S3 API, it reports 500, but the REST API can be adjusted. 5ae38ee4f679f9421d983aa0f6d9a832

jasondrogba commented 4 weeks ago

Do you look at api-proxy in fluid? https://github.com/fluid-cloudnative/fluid/blob/e8eff84c89f9a1453b9c59a00ba07c6180a474fd/docs/en/samples/api_proxy.md

It seem that your request is illegal, you can look doc rest-api and s3-api to interact with Alluxio through REST API or S3 API

lgy1027 commented 3 weeks ago

Do you look at api-proxy in fluid? https://github.com/fluid-cloudnative/fluid/blob/e8eff84c89f9a1453b9c59a00ba07c6180a474fd/docs/en/samples/api_proxy.md

It seem that your request is illegal, you can look doc rest-api and s3-api to interact with Alluxio through REST API or S3 API

The rest api can be interacted with, but the s3 api cannot be accessed. I am not sure if there is a problem with the Authorization settings. Can you give me some help? According to the official documents, I don't understand how to set the Authorization value.

lgy1027 commented 3 weeks ago

Do you look at api-proxy in fluid? https://github.com/fluid-cloudnative/fluid/blob/e8eff84c89f9a1453b9c59a00ba07c6180a474fd/docs/en/samples/api_proxy.md It seem that your request is illegal, you can look doc rest-api and s3-api to interact with Alluxio through REST API or S3 API

The rest api can be interacted with, but the s3 api cannot be accessed. I am not sure if there is a problem with the Authorization settings. Can you give me some help? According to the official documents, I don't understand how to set the Authorization value.

alluxio-master apigateway err log:

2024-06-12 01:38:36,659 INFO  RequestLoggingFilter - Alluxio S3 API received GET request: URI=http://demo-master-0.default:25653/api/v1/s3/test User=AWS4-HMAC-SHA256 Credential=app-6643791379886311122/... SignedHeaders=... Signature=... Media Type=null Query Parameters={} Path Parameters={}
Jun 12, 2024 1:38:36 AM org.glassfish.jersey.internal.Errors logErrors
WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 2
alluxio.exception.AlluxioException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at alluxio.exception.status.AlluxioStatusException.toAlluxioException(AlluxioStatusException.java:110)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:606)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:151)
        at alluxio.client.file.MetadataCachingFileSystem.createDirectory(MetadataCachingFileSystem.java:99)
        at alluxio.proxy.s3.S3RestServiceHandler.<init>(S3RestServiceHandler.java:157)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356)
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at alluxio.web.ProxyWebServer$1.service(ProxyWebServer.java:109)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
2024-06-12 01:38:36,706 WARN  HttpChannel - handleException /api/v1/s3/test alluxio.exception.status.UnknownException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:750)
Caused by: alluxio.exception.status.UnknownException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at alluxio.exception.status.AlluxioStatusException.from(AlluxioStatusException.java:174)
        at alluxio.exception.status.AlluxioStatusException.fromStatusRuntimeException(AlluxioStatusException.java:215)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:483)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:447)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:436)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:158)
        at alluxio.client.file.BaseFileSystem.lambda$createDirectory$1(BaseFileSystem.java:154)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:591)
        ... 70 more
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
        at alluxio.grpc.FileSystemMasterClientServiceGrpc$FileSystemMasterClientServiceBlockingStub.createDirectory(FileSystemMasterClientServiceGrpc.java:1873)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.lambda$createDirectory$2(RetryHandlingFileSystemMasterClient.java:159)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:481)
        ... 75 more
MultiException stack 2 of 2
java.lang.IllegalStateException: Unable to perform operation: create on alluxio.proxy.s3.S3RestServiceHandler
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:369)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at alluxio.web.ProxyWebServer$1.service(ProxyWebServer.java:109)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:750)

2024-06-12 01:38:36,706 WARN  HttpChannelState - unhandled due to prior sendError
javax.servlet.ServletException: A MultiException has 2 exceptions.  They are:
1. alluxio.exception.AlluxioException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
2. java.lang.IllegalStateException: Unable to perform operation: create on alluxio.proxy.s3.S3RestServiceHandler

        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:410)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at alluxio.web.ProxyWebServer$1.service(ProxyWebServer.java:109)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:750)
Caused by: A MultiException has 2 exceptions.  They are:
1. alluxio.exception.AlluxioException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
2. java.lang.IllegalStateException: Unable to perform operation: create on alluxio.proxy.s3.S3RestServiceHandler

        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:368)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        ... 31 more
Caused by: alluxio.exception.AlluxioException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at alluxio.exception.status.AlluxioStatusException.toAlluxioException(AlluxioStatusException.java:110)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:606)
        at alluxio.client.file.BaseFileSystem.createDirectory(BaseFileSystem.java:151)
        at alluxio.client.file.MetadataCachingFileSystem.createDirectory(MetadataCachingFileSystem.java:99)
        at alluxio.proxy.s3.S3RestServiceHandler.<init>(S3RestServiceHandler.java:157)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.makeMe(ReflectionHelper.java:1356)
        at org.jvnet.hk2.internal.ClazzCreator.createMe(ClazzCreator.java:248)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:342)
        ... 60 more
Caused by: alluxio.exception.status.UnknownException: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at alluxio.exception.status.AlluxioStatusException.from(AlluxioStatusException.java:174)
        at alluxio.exception.status.AlluxioStatusException.fromStatusRuntimeException(AlluxioStatusException.java:215)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:483)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:447)
        at alluxio.AbstractClient.retryRPC(AbstractClient.java:436)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.createDirectory(RetryHandlingFileSystemMasterClient.java:158)
        at alluxio.client.file.BaseFileSystem.lambda$createDirectory$1(BaseFileSystem.java:154)
        at alluxio.client.file.BaseFileSystem.rpc(BaseFileSystem.java:591)
        ... 70 more
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: Cannot create or load UFS directory /underFSStorage/.alluxio_s3_api_metadata: java.io.FileNotFoundException: /underFSStorage/.alluxio_s3_api_metadata.
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
        at alluxio.grpc.FileSystemMasterClientServiceGrpc$FileSystemMasterClientServiceBlockingStub.createDirectory(FileSystemMasterClientServiceGrpc.java:1873)
        at alluxio.client.file.RetryHandlingFileSystemMasterClient.lambda$createDirectory$2(RetryHandlingFileSystemMasterClient.java:159)
        at alluxio.AbstractClient.retryRPCInternal(AbstractClient.java:481)
        ... 75 more