Failed to connect to the Ceph cluster

[pareshpp]

hi, I have a single node ceph cluster on ubuntu where i have installed cosbench 0.4.2.c2.

I am getting error as "Failed to connect to the Ceph cluster" as shown below.

Workload file ================================================================== <?xml version="1.0" encoding="UTF-8" ?>

``` ``` ``` ```

cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQAKzJNWJa7tIRAApeMbgrIf3d2XwxNRrLGK0g==

=================================================system.log 2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - begin to process workload w29 2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - begin to run stage s1-prepare 2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - ============================================ 2016-01-13 20:04:33,416 [INFO] [WorkloadProcessor] - START WORK: prepare 2016-01-13 20:04:33,029 [INFO] [AbstractCommandTasklet] - time drift between controller and driver-driver1 is 29 mSec 2016-01-13 20:04:33,029 [INFO] [StageRunner] - successfully booted all tasks in stage s1-prepare 2016-01-13 20:09:33,123 [ERROR] [AbstractCommandHandler] - unexpected error com.intel.cosbench.api.storage.StorageException: com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110) at com.intel.cosbench.api.librados.LibradosStorage.init(LibradosStorage.java:96) at com.intel.cosbench.driver.service.COSBStorageAPIService.getStorage(COSBStorageAPIService.java:54) at com.intel.cosbench.driver.service.MissionHandler.createStorageApi(MissionHandler.java:210) at com.intel.cosbench.driver.service.MissionHandler.createWorkerContext(MissionHandler.java:197) at com.intel.cosbench.driver.service.MissionHandler.createWorkers(MissionHandler.java:185) at com.intel.cosbench.driver.service.MissionHandler.init(MissionHandler.java:100) at com.intel.cosbench.driver.service.COSBDriverService.createHandler(COSBDriverService.java:112) at com.intel.cosbench.driver.service.COSBDriverService.submit(COSBDriverService.java:83) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:71) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:54) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy20.submit(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:71) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:54) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy5.submit(Unknown Source) at com.intel.cosbench.driver.handler.SubmitHandler.process(SubmitHandler.java:38) at com.intel.cosbench.driver.handler.AbstractCommandHandler.handleRequest(AbstractCommandHandler.java:76) at com.intel.cosbench.driver.handler.SubmitHandler.handleRequest(SubmitHandler.java:1) at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:560) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:745) Caused by: com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110) at com.ceph.rados.RadosBase.throwException(RadosBase.java:87) at com.ceph.rados.RadosBase.handleReturnCode(RadosBase.java:50) at com.ceph.rados.Rados.connect(Rados.java:150) at com.intel.cosbench.api.librados.LibradosStorage.init(LibradosStorage.java:76) ... 62 more 2016-01-13 20:09:33,126 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 500 - com.ceph.rados.exceptions.RadosTimeoutException: Failed to connect to the Ceph cluster; ETIMEDOUT: Connection timed out (-110) 2016-01-13 20:09:33,127 [ERROR] [StageRunner] - detected tasks [t1] have encountered errors 2016-01-13 20:09:33,127 [INFO] [StageRunner] - begin to terminate stage s1-prepare 2016-01-13 20:09:33,132 [ERROR] [AbstractCommandTasklet] - driver report error: HTTP 400 - unrecognized request: org.apache.catalina.connector.RequestFacade@277c2b4e 2016-01-13 20:09:33,132 [ERROR] [Aborter] - fail to abort driver com.intel.cosbench.controller.tasklet.TaskletException at com.intel.cosbench.controller.tasklet.AbstractCommandTasklet.issueCommand(AbstractCommandTasklet.java:81) at com.intel.cosbench.controller.tasklet.Aborter.executeAbort(Aborter.java:53) at com.intel.cosbench.controller.tasklet.Aborter.execute(Aborter.java:42) at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:47) at com.intel.cosbench.controller.tasklet.AbstractTasklet.call(AbstractTasklet.java:1) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-01-13 20:09:33,133 [INFO] [StageRunner] - stage s1-prepare has been terminated 2016-01-13 20:09:33,133 [ERROR] [WorkloadProcessor] - detected stage s1-prepare encountered error 2016-01-13 20:09:33,133 [INFO] [WorkloadProcessor] - begin to terminate workload w29 2016-01-13 20:09:33,196 [INFO] [SimpleWorkloadArchiver] - workload w29 has been successfully archived 2016-01-13 20:09:33,196 [INFO] [WorkloadProcessor] - successfully terminated workload w29

[pareshpp]

I could resolve this issue by using public address in workload file. Thanks Anyways.

[zonArt]

Hi,

I've got the exact same issue using rados-java library. It's like it's willing to go out on a proxy (for my case) and I guess that's why you mention the public address (even if I don't really understand where you set it). Strangely this is working on one server I'm using it and on the other one not. If you could explain me the hypothesis on what's going on, I'd appreciate a lot. Thank you