search

JaneliaSciComp / jacs-cm

JACS Configuration Management
Other
8 stars 3 forks source link

Import Imagery with Docker Compose Deployment #10

Closed freshpomelo closed 4 years ago

freshpomelo commented 4 years ago

Hi all,

I've install jacs-cm in the "Docker Compose Deployment" mode instead of swarm mode.

And workstation have connected to jacs-cm successfully.

Following the explanation in https://github.com/JaneliaSciComp/jacs-cm/issues/8#issuecomment-546625985, I've create my own user and add to the relative groups.

image

When I tried to import imagery from workstation, there was an error with 401 code:

2019-11-21_01:15:15.119 [XNIO-1 task-48] INFO o.janelia.jacs2.app.JAXSyncAppConfig [service=] [rootService=] - 172.30.0.14 youmin youmin 'Workstation' GET http://localhost/api/rest-v2/data/domainobject/class ?subjectKey=user%3Ayoumin&domainClass=org.janelia.model.domain.gui.cdmip.ColorDepthLibrary status=200 response_bytes=2 rt=0.09 tp=0.002 'Jersey/2.21 (HttpUrlConnection 1.8.0_231)
2019-11-21_01:15:15.133 [XNIO-1 task-49] INFO o.janelia.jacs2.app.JAXSyncAppConfig [service=] [rootService=] - 172.30.0.14 youmin youmin 'Workstation' GET http://localhost/api/rest-v2/process/release ?subjectKey=user%3Ayoumin status=200 response_bytes=2 rt=0.09 tp=0.002 'Jersey/2.21 (HttpUrlConnection 1.8.0_231)
2019-11-21_01:15:15.154 [XNIO-1 task-50] INFO o.janelia.jacs2.app.JAXSyncAppConfig [service=] [rootService=] - 172.30.0.14 youmin youmin 'Workstation' GET http://localhost/api/rest-v2/data/dataset ?subjectKey=user%3Ayoumin status=200 response_bytes=2 rt=0.08 tp=0.002 'Jersey/2.21 (HttpUrlConnection 1.8.0_231)
2019-11-21_01:15:41.513 [XNIO-1 task-51] INFO o.j.j.r.s.v.d.TmSampleResource [service=] [rootService=] - Creating new TmSample dulab_test with path /jade1/dulab_test
2019-11-21_01:15:41.677 [XNIO-1 task-51] WARN o.j.j.d.storage.StorageService [service=] [rootService=] - Request JerseyWebTarget { http://jade-master:8080/jacsstorage/master_api/v1/storage?storagePath=%2Fjade1%2Fdulab_test&ownerKey=user%3Ayoumin } returned status 401 while trying to get the storage for storageId = null, storageName=null, storagePath=/jade1/dulab_test 2019-11-21_01:15:41.681 [XNIO-1 task-51] ERROR o.j.j.r.IllegalStateRequestHandler [service=] [rootService=] - Invalid state exception java.lang.IllegalStateException: Invalid response received from http://jade-master:8080/jacsstorage/master_api/v1/storage?storagePath=%2Fjade1%2Fdulab_test&ownerKey=user%3Ayoumin while trying to request the storage URL at
org.janelia.jacs2.dataservice.storage.StorageService.lookupDataStorage(StorageService.java:95)
at org.janelia.jacs2.dataservice.storage.DataStorageLocationFactory.lookupJadeDataLocation(DataStorageLocationFactory.java:56) 
at org.janelia.jacs2.rest.sync.v2.dataresources.TmSampleResource.createTmSample(TmSampleResource.java:240) 
at org.janelia.jacs2.rest.sync.v2.dataresources.TmSampleResource$Proxy$_$$_WeldClientProxy.createTmSample(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) 
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) 
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) 
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) 
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) 
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) 
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) 
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) 
at org.glassfish.jersey.internal.Errors.process(Errors.java:292) 
at org.glassfish.jersey.internal.Errors.process(Errors.java:274) 
at org.glassfish.jersey.internal.Errors.process(Errors.java:244) 
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) 
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) 
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679) 
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:392) 
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) 
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:365) 
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:318) 
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) 
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) 
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) 
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) 
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) 
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) 
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) 
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) 
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) 
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) 
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) 
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) 
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) 
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) 
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) 
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) 
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) 
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) 
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) 
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) 
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) 
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:364) 
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748)
2019-11-21_01:15:41.691 [XNIO-1 task-51] INFO o.janelia.jacs2.app.JAXSyncAppConfig [service=] [rootService=] - 172.30.0.14 youmin youmin 'Workstation' PUT http://localhost/api/rest-v2/mouselight/data/sample ?subjectKey=user%3Ayoumin status=500 response_bytes=204 rt=0.07 tp=0.008 'Jersey/2.21 (HttpUrlConnection 1.8.0_231)' {"subjectKey":"user:youmin","references":null,"objectIds":null,"ordering":null,"objectType":null,"propertyName":null,"propertyValue":null,"domainObject":{"class":"org.janelia.model.domain.tiledMicroscope.TmSample","name":"dulab_test","ownerKey":null,"readers":[],"writers":[],"creationDate":null,"updatedDate":null,"micronToVoxMatrix":null,"voxToMicronMatrix":null,"origin":null,"scaling":null,"numImageryLevels":null,"files":{"LargeVolumeOctree":"/jade1/dulab_test"},"filesystemSync":true,"filepath":null,"_id":... truncated ...

Any advice would be greatly appreciated! Thanks a lot in advance.

freshpomelo commented 4 years ago

And this is the error log in jac-storage:

2019-11-26T02:20:48.094921100Z 2019-11-26_02:20:48.067 [XNIO-1 task-1] INFO  o.j.j.rest.StorageAgentsResource - Register agent - org.janelia.jacsstorage.datarequest.StorageAgentInfo@2fbc0cba[agentId=localhost:9881,agentAccessURL=http://localhost:9881/jacsstorage/agent_api/v1,agentToken=<null>,connectionStatus=DISCONNECTED]
2019-11-26T02:20:48.384427800Z 2019-11-26_02:20:48.384 [XNIO-1 task-1] INFO  o.j.j.cdi.PersistenceProducer - Use replica set: rsJacs
2019-11-26T02:20:48.386818700Z 2019-11-26_02:20:48.386 [XNIO-1 task-1] INFO  o.j.j.cdi.PersistenceProducer - Connect to MongoDB (jade@mongo1:27017,mongo2:27017,mongo3:27017)
2019-11-26T02:20:49.028933400Z 2019-11-26_02:20:49.028 [XNIO-1 task-1] INFO  o.j.j.app.JAXMasterStorageApp - 192.168.0.14 unknown unknown '' POST http://jade-master:8080/jacsstorage/master_api/v1/agents  status=201 response_bytes=199 rt=2.480 tp=0.001 'Jersey/2.29.1 (HttpUrlConnection 1.8.0_232)' {"agentId":"localhost:9881","agentAccessURL":"http://localhost:9881/jacsstorage/agent_api/v1","servedVolumes":["*"],"unavailableVolumeIds":[],"connectionStatus":"DISCONNECTED","agentToken":null}
2019-11-26T02:21:19.324933400Z 2019-11-26_02:21:19.323 [JACS-STORAGE-TASK-0] WARN  o.j.j.s.d.AgentConnectionHelper - Error raised during agent getStatus
2019-11-26T02:21:19.324971800Z javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
2019-11-26T02:21:19.325017800Z  at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:260)
2019-11-26T02:21:19.325037800Z  at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:254)
2019-11-26T02:21:19.325056300Z  at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:736)
2019-11-26T02:21:19.325095600Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
2019-11-26T02:21:19.325114500Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
2019-11-26T02:21:19.325133000Z  at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
2019-11-26T02:21:19.325150600Z  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
2019-11-26T02:21:19.325169400Z  at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:735)
2019-11-26T02:21:19.325187800Z  at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:395)
2019-11-26T02:21:19.325219600Z  at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:295)
2019-11-26T02:21:19.325238000Z  at org.janelia.jacsstorage.service.distributedservice.AgentConnectionHelper.getAgentStatus(AgentConnectionHelper.java:31)
2019-11-26T02:21:19.325256500Z  at org.janelia.jacsstorage.service.distributedservice.AgentConnectionTester.testConnection(AgentConnectionTester.java:11)
2019-11-26T02:21:19.325274600Z  at org.janelia.jacsstorage.service.distributedservice.AgentConnectionTester.testConnection(AgentConnectionTester.java:7)
2019-11-26T02:21:19.325292600Z  at org.janelia.jacsstorage.resilience.PeriodicConnectionChecker.lambda$initialize$0(PeriodicConnectionChecker.java:39)
2019-11-26T02:21:19.325310600Z  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2019-11-26T02:21:19.325328500Z  at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2019-11-26T02:21:19.325346400Z  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2019-11-26T02:21:19.325364400Z  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2019-11-26T02:21:19.325384400Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-11-26T02:21:19.325408700Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-11-26T02:21:19.325430500Z  at java.lang.Thread.run(Thread.java:748)
2019-11-26T02:21:19.325470000Z Caused by: java.net.ConnectException: Connection refused (Connection refused)
2019-11-26T02:21:19.325490200Z  at java.net.PlainSocketImpl.socketConnect(Native Method)
2019-11-26T02:21:19.325511300Z  at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
2019-11-26T02:21:19.325530700Z  at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
2019-11-26T02:21:19.325553600Z  at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
2019-11-26T02:21:19.325575500Z  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2019-11-26T02:21:19.325597700Z  at java.net.Socket.connect(Socket.java:607)
2019-11-26T02:21:19.325618000Z  at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
2019-11-26T02:21:19.325639900Z  at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
2019-11-26T02:21:19.325660800Z  at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
2019-11-26T02:21:19.325678900Z  at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
2019-11-26T02:21:19.325700300Z  at sun.net.www.http.HttpClient.New(HttpClient.java:339)
2019-11-26T02:21:19.325716100Z  at sun.net.www.http.HttpClient.New(HttpClient.java:357)
2019-11-26T02:21:19.325734200Z  at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
2019-11-26T02:21:19.325748700Z  at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162)
2019-11-26T02:21:19.325769000Z  at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
2019-11-26T02:21:19.325782300Z  at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
2019-11-26T02:21:19.325800500Z  at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570)
2019-11-26T02:21:19.325818400Z  at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
2019-11-26T02:21:19.325832800Z  at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
2019-11-26T02:21:19.325851300Z  at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:366)
2019-11-26T02:21:19.325869100Z  at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:258)
2019-11-26T02:21:19.325886900Z  ... 20 common frames omitted
krokicki commented 4 years ago

We may be able to improve the error handling here, and we'll look into that.

But the bottom line is that your JADE agent is not accessible somehow. It's some kind of network communication issue. Please check if the JADE agent is running using the troubleshooting steps.

cgoina commented 4 years ago

The 401 actually happens in jacs-compute most likely because the api key to connect to JADE is not configured properly. The jacs agent connectivity has nothing to do with the 401.

cgoina commented 4 years ago

Make sure the value for StorageService.ApiKey= in your jacs.properties matches the value of StorageService.ApiKey from jade configuration. The latter gets set from JADE_API_KEY environment variable set in your .env.config